Methods and apparatus for adjusting a level of an audio signal

ABSTRACT

The invention relates to methods and apparatus for adjusting a level of an audio signal. An audio signal is divided into a plurality of frequency bands. Modification parameters are obtained for at least one of the plurality of frequency band. Gain factors are derived for at least one of the plurality of frequency bands, the gain factors determined based on the amplitude scale factors. The gain factors are smoothed. A level of noise from noise compensation factors is determined. The gain factors are applied to at least one of the frequency bands to generate gain adjusted frequency bands. The level of noise is adjusted based on the gain adjusted frequency bands. At least one of the frequency bands is filtered with a filter generated with the filter coefficients. The plurality of frequency bands is synthesized to generate an output audio signal.

CROSS-REFERENCE TO RELATED APPLICATION; BENEFIT CLAIM

This application is a divisional application which claims benefit toU.S. Continuation application Ser. No. 15/663,558, filed Jul. 28, 2017,which claims benefit of U.S. Continuation application Ser. No.15/431,663 filed Feb. 13, 2017, now issued as U.S. Pat. No. 9,768,748,issued Aug. 30, 2017, which claims benefit of U.S. Continuationapplication Ser. No. 15/143,501, filed Apr. 29, 2016, now issued as U.S.Pat. No. 9,608,584 dated Mar. 28, 2017 which claims benefit as a U.S.Continuation of application Ser. No. 13/919,891, filed Jun. 17, 2013,now U.S. Pat. No. 9,350,311, issued May 24, 2016, which claims benefitas a U.S. Continuation of application Ser. No. 13/338,126, filed Dec.27, 2011, now U.S. Pat. No. 8,488,809, issued Jul. 16, 2013, whichclaims benefit as a Continuation of application Ser. No. 11/666,252,filed Apr. 25, 2007, now U.S. Pat. No. 8,090,120, issued Jan. 3, 2012,which is a National Stage Application under 35 U.S.C. § 371 ofInternational Application No. PCT/US2005/38579, filed Oct. 25, 2005,which claims benefit of Provisional Application No. 60/638,607 filedDec. 21, 2004, and also claims benefit of Provisional Application No.60/622,458 filed Oct. 26, 2004, the entire contents of all of which arehereby incorporated by reference as if fully set forth herein, under 35U.S.C. § 120. The applicant(s) hereby rescind any disclaimer of claimscope in the parent application(s) or the prosecution history thereofand advise the USPTO that the claims in this application may be broaderthan any claim in the parent application(s).

TECHNICAL FIELD

The invention relates to audio signal processing. More particularly, theinvention relates to the measurement and control of the perceived soundloudness and/or the perceived spectral balance of an audio signal. Theinvention is useful, for example, in one or more of:loudness-compensating volume control, automatic gain control, dynamicrange control (including, for example, limiters, compressors, expanders,etc.), dynamic equalization, and compensating for background noiseinterference in an audio playback environment. The invention includesnot only methods but also corresponding computer programs and apparatus.

BACKGROUND ART

There have been many attempts to develop a satisfactory objective methodof measuring loudness. Fletcher and Munson determined in 1933 that humanhearing is less sensitive at low and high frequencies than at middle (orvoice) frequencies. They also found that the relative change insensitivity decreased as the level of the sound increased. An earlyloudness meter consisted of a microphone, amplifier, meter and acombination of filters designed to roughly mimic the frequency responseof hearing at low, medium and high sound levels.

Even though such devices provided a measurement of the loudness of asingle, constant level, isolated tone, measurements of more complexsounds did not match the subjective impressions of loudness very well.Sound level meters of this type have been standardized but are only usedfor specific tasks, such as the monitoring and control of industrialnoise.

In the early 1950s, Zwicker and Stevens, among others, extended the workof Fletcher and Munson in developing a more realistic model of theloudness perception process. Stevens published a method for the“Calculation of the Loudness of Complex Noise” in the Journal of theAcoustical Society of America in 1956, and Zwicker published his“Psychological and Methodical Basis of Loudness” article in Acoustica in1958. In 1959 Zwicker published a graphical procedure for loudnesscalculation, as well as several similar articles shortly after. TheStevens and Zwicker methods were standardized as ISO 532, parts A and B(respectively). Both methods involve similar steps.

First, the time-varying distribution of energy along the basilarmembrane of the inner ear, referred to as the excitation, is simulatedby passing the audio through a bank of band-pass auditory filters withcenter frequencies spaced uniformly on a critical band rate scale. Eachauditory filter is designed to simulate the frequency response at aparticular location along the basilar membrane of the inner ear, withthe filter's center frequency corresponding to this location. Acritical-band width is defined as the bandwidth of one such filter.Measured in units of Hertz, the critical-band width of these auditoryfilters increases with increasing center frequency. It is thereforeuseful to define a warped frequency scale such that the critical-bandwidth for all auditory filters measured in this warped scale isconstant. Such a warped scale is referred to as the critical band ratescale and is very useful in understanding and simulating a wide range ofpsychoacoustic phenomena. See, for example, Psychoacoustics—Facts andModels by E. Zwicker and H. Fastl, Springer-Verlag, Berlin, 1990. Themethods of Stevens and Zwicker utilize a critical band rate scalereferred to as the Bark scale, in which the critical-band width isconstant below 500 Hz and increases above 500 Hz. More recently, Mooreand Glasberg defined a critical band rate scale, which they named theEquivalent Rectangular Bandwidth (ERB) scale (B. C. J. Moore, B.Glasberg, T. Baer, “A Model for the Prediction of Thresholds, Loudness,and Partial Loudness,” Journal of the Audio Engineering Society, Vol.45, No. 4, April 1997, pp. 224-240). Through psychoacoustic experimentsusing notched-noise maskers, Moore and Glasberg demonstrated that thecritical-band width continues to decrease below 500 Hz, in contrast tothe Bark scale where the critical-band width remains constant.

Following the computation of excitation is a non-linear compressivefunction that generates a quantity referred to as “specific loudness”.Specific loudness is a measure of perceptual loudness as a function offrequency and time and may be measured in units of perceptual loudnessper unit frequency along a critical band rate scale, such as the Bark orERB scale discussed above. Finally, the time-varying “total loudness” iscomputed by integrating specific loudness across frequency. Whenspecific loudness is estimated from a finite set of auditory filtersdistributed uniformly along a critical band rate scale, total loudnessmay be computed by simply summing the specific loudness from eachfilter.

Loudness may be measured in units of phon. The loudness of a given soundin phon is the sound pressure level (SPL) of a 1 kHz tone having asubjective loudness equal to that of the sound. Conventionally, thereference 0 dB for SPL is a root mean square pressure of 2×10⁻⁵ Pascal,and this is also therefore the reference 0 phon. Using this definitionin comparing the loudness of tones at frequencies other than 1 kHz withthe loudness at 1 kHz, a contour of equal loudness can be determined fora given phon level. FIG. 11 shows equal loudness contours forfrequencies between 20 Hz and 12.5 kHz, and for phon levels between 4.2phon (considered to be the threshold of hearing) and 120 phon (ISO226:1087 (E), “Acoustics—Normal equal loudness level contours”). The phonmeasurement takes into account the varying sensitivity of human hearingwith frequency, but the results do not allow the assessment of therelative subjective loudnesses of sounds at varying levels because thereis no attempt to correct for the non-linearity of the growth of loudnesswith SPL, that is, for the fact that the spacing of the contours varies.

Loudness may also be measured in units of “sone”. There is a one-to-onemapping between phon units and sone units, as indicated in FIG. 11. Onesone is defined as the loudness of a 40 dB (SPL) 1 kHz pure sine waveand is equivalent to 40 phon. The units of sone are such that a twofoldincrease in sone corresponds to a doubling of perceived loudness. Forexample, 4 sone is perceived as twice as loud as 2 sone. Thus,expressing loudness levels in sone is more informative. Given thedefinition of specific loudness as a measure of perceptual loudness as afunction of frequency and time, specific loudness may be measured inunits of sone per unit frequency. Thus, when using the Bark scale,specific loudness has units of sone per Bark and likewise when using theERB scale, the units are sone per ERB.

As mentioned above, the sensitivity of the human ear varies with bothfrequency and level, a fact well documented in the psychoacousticsliterature. One of the results is that the perceived spectrum or timbreof a given sound varies with the acoustic level at which the sound isheard. For example, for a sound containing low, middle and highfrequencies, the perceived relative proportions of such frequencycomponents change with the overall loudness of the sound; when it isquiet the low and high frequency components sound quieter relative tothe middle frequencies than they sound when it is loud. This phenomenonis well known and has been mitigated in sound reproducing equipment byso-called loudness controls. A loudness control is a volume control thatapplies low- and sometimes also high-frequency boost as the volume isturned down. Thus, the lower sensitivity of the ear at the frequencyextremes is compensated by an artificial boost of those frequencies.Such controls are completely passive; the degree of compensation appliedis a function of the setting of the volume control or some otheruser-operated control, not as a function of the content of the audiosignals.

In practice, changes in perceived relative spectral balance among low,middle and high frequencies depend on the signal, in particular on itsactual spectrum and on whether it is intended to be loud or soft.Consider the recording of a symphony orchestra. Reproduced at the samelevel that a member of the audience would hear in a concert hall, thebalance across the spectrum may be correct whether the orchestra isplaying loudly or quietly. If the music is reproduced 10 dB quieter, forexample, the perceived balance across the spectrum changes in one mannerfor loud passages and changes in another manner for quiet passages. Aconventional passive loudness control does not apply differentcompensations as a function of the music.

In International Patent Application No. PCT/US2004/016964, filed May 27,2004, published Dec. 23, 2004 as WO 2004/111994 A2, Seefeldt et aldisclose, among other things, a system for measuring and adjusting theperceived loudness of an audio signal. Said PCT application, whichdesignates the United States, is hereby incorporated by reference in itsentirety. In said application, a psychoacoustic model calculates theloudness of an audio signal in perceptual units. In addition, theapplication introduces techniques for computing a widebandmultiplicative gain, which, when applied to the audio, results in theloudness of the gain-modified audio being substantially the same as areference loudness. Application of such wideband gain, however, changesthe perceived spectral balance of the audio.

DISCLOSURE OF THE INVENTION

In one aspect, the invention provides for deriving information usablefor controlling the specific loudness of an audio signal by modifyingthe audio signal in order to reduce the difference between its specificloudness and a target specific loudness. Specific loudness is a measureof perceptual loudness as a function of frequency and time. In practicalimplementations, the specific loudness of the modified audio signal maybe made to approximate the target specific loudness. The approximationmay be affected not only by ordinary signal processing considerationsbut also time- and/or frequency-smoothing that may be employed in themodifying, as described below.

Because specific loudness is a measure of perceptual loudness of anaudio signal as a function of frequency and time, in order to reduce thedifference between the specific loudness of the audio signal and thetarget specific loudness, the modifying may modify the audio signal as afunction of frequency. Although in some cases the target specificloudness may be time-invariant and the audio signal itself may be asteady-state time-invariant signal, typically, the modifying may alsomodify the audio signal as a function of time.

Aspects of the present invention may also be employed to compensate forbackground noise interfering in an audio playback environment. Whenaudio is heard in the presence of background noise, the noise maypartially or completely mask the audio in a manner dependent on both thelevel and spectrum of the audio and the level and spectrum of the noise.The result is an alteration in the perceived spectrum of the audio. Inaccordance with psychoacoustic studies (see, for example, Moore,Glasberg, and Baer, “A Model for the Prediction of Thresholds, Loudness,and Partial Loudness,” J. Audio Eng. Soc., Vol. 45, No. 4, April 1997),one may define the “partial specific loudness” of the audio as theperceptual loudness of the audio in the presence of a secondaryinterfering sound signal, such as the noise.

Thus, in another aspect, the invention provides for deriving informationusable for controlling the partial specific loudness of an audio signalby modifying the audio signal in order to reduce the difference betweenits partial specific loudness and a target specific loudness. Doing somitigates the effects of the noise in a perceptually accurate manner. Inthis and other aspects of the invention that take an interfering noisesignal into account, it is assumed that there is access to the audiosignal by itself and the secondary interfering signal by itself.

In another aspect, the invention provides for controlling the specificloudness of an audio signal by modifying the audio signal in order toreduce the difference between its specific loudness and a targetspecific loudness.

In another aspect, the invention provides for controlling the partialspecific loudness of an audio signal by modifying the audio signal inorder to reduce the difference between its partial specific loudness anda target specific loudness.

When the target specific loudness is not a function of the audio signal,it may be a stored or received target specific loudness. When the targetspecific loudness is not a function of the audio signal, the modifyingor the deriving may explicitly or implicitly calculate specific loudnessor partial specific loudness. Examples of implicit calculation include,a lookup table or a “closed-form” mathematical expression, in whichspecific loudness and/or partial specific loudness is inherentlydetermined (the term closed-form is meant to describe a mathematicalexpression which can be represented exactly using a finite number ofstandard mathematical operations and functions, such as exponentiationand cosine). Also when the target specific loudness is not a function ofthe audio signal, the target specific loudness may be both time- andfrequency-invariant or it may be only time-invariant.

In yet another aspect, the invention provides for processing an audiosignal by processing the audio signal or a measure of the audio signalin accordance with one or more processes and one or moreprocess-controlling parameters to produce a target specific loudness.Although the target specific loudness may be time-invariant (“fixed”),the target specific loudness may advantageously be a function of thespecific loudness of the audio signal. Although it may be a static,frequency- and time-invariant signal, typically, the audio signal itselfis frequency- and time-varying, thus causing the target specificloudness to be frequency- and time-varying when it is a function of theaudio signal.

The audio and a target specific loudness or a representation of a targetspecific loudness may be received from a transmission or reproduced froma storage medium.

The representation of a target specific loudness may be one or morescale factors that scale the audio signal or measure of the audiosignal.

The target specific loudness of any of the above aspects of theinvention may be a function of the audio signal or measure of the audiosignal. One suitable measure of the audio signal is the specificloudness of the audio signal. The function of the audio signal ormeasure of the audio signal may be a scaling of the audio signal ormeasure of the audio signal. For example, the scaling may be one or acombination of scalings:

(a) a time- and frequency-varying scale factor Ξ[b,t] scaling of thespecific loudness as in the relationship{circumflex over (N)}[b,t]=Ξ[b,t]N[b,t];

(b) a time-varying, frequency-invariant scale factor Φ[t] scaling of thespecific loudness as in the relationship{circumflex over (N)}[b,t]=Φ[t]N[b,t];

(c) a time-invariant, frequency-varying scale factor Θ[b] scaling of thespecific loudness as in the relationship{circumflex over (N)}[b,t]=Θ[b]N[b,t];and

(d) a time-invariant, frequency-invariant, scale factor α scaling of thespecific loudness of the audio signal as in the relationship{circumflex over (N)}[b,t]=αN[b,t],wherein {circumflex over (N)}[b,t] is the target specific loudness,N[b,t] is the specific loudness of the audio signal, b is a measure offrequency, and t is a measure of time.

In the case (a) of a time- and frequency-varying scale factor, thescaling may be determined at least in part by a ratio of a desiredmultiband loudness and the multiband loudness of the audio signal. Sucha scaling may be usable as a dynamic range control. Further details ofemploying aspects of the invention as a dynamic range control are setforth below.

Also in the case (a) of a time- and frequency-varying scale factor, thespecific loudness may scaled by the ratio of a measure of a desiredspectral shape to the measure of a spectral shape of the audio signal.Such a scaling may be employed to transform the perceived spectrum ofthe audio signal from a time-varying perceived spectrum to asubstantially time-invariant perceived spectrum. When the specificloudness is scaled by the ratio of a measure of a desired spectral shapeto the measure of a spectral shape of the audio signal, such a scalingmay be usable as a dynamic equalizer. Further details of employingaspects of the invention as a dynamic equalizer are set forth below.

In the case (b) of a time-varying, frequency-invariant scale factor, thescaling may be determined at least in part by a ratio of a desiredwideband loudness and the wideband loudness of the audio signal. Such ascaling may be usable as an automatic gain control or dynamic rangecontrol. Further details of employing aspects of the invention as anautomatic gain control or a dynamic range control are set forth below

In case (a) (a time- and frequency-varying scale factor) or case (b) (atime-varying, frequency-invariant scale factor), the scale factor may bea function of the audio signal or measure of the audio signal.

In both the case (c) of a time-invariant, frequency-varying scale factoror the case (d) of a time-invariant, frequency-invariant, scale factor,the modifying or the deriving may include storing the scale factor orthe scale factor may be received from an external source.

In either of cases (c) and (d), the scale factor may not be a functionof the audio signal or measure of the audio signal.

In any of the various aspects of the invention and the variationsthereof, the modifying, deriving, or producing may, variously,explicitly or implicitly calculate (1) specific loudness, and/or (2)partial specific loudness, and/or (3) the target specific loudness.Implicit calculations may involve, for example, a lookup table or aclosed-form mathematical expression.

Modification parameters may be temporally smoothed. Modificationparameters may be, for example, (1) a plurality of amplitude scalingfactors relating to frequency bands of the audio signal or (2) aplurality of filter coefficients for controlling one or more filters,such as a multitapped FIR filter or a multipole IIR filter. The scalingfactors or filter coefficients (and the filters to which they areapplied) may be time-varying.

In calculating the function of the specific loudness of the audio signalthat defines the target specific loudness or the inverse of thatfunction, the process or processes performing such calculations operatesin what may be characterized as the perceptual (psychoacoustic) loudnessdomain—the input and output of the calculation are specific loudnesses.In contrast, in applying amplitude scaling factors to frequency bands ofthe audio signal or applying filter coefficients to a controllablefiltering of the audio signal, the modification parameters operate tomodify the audio signal outside the perceptual (psychoacoustic) loudnessdomain in what may be characterized as the electrical signal domain.Although modifications to the audio signal may be made to the audiosignal in the electrical signal domain, such changes in the electricalsignal domain are derived from calculations in the perceptual(psychoacoustic) loudness domain such that the modified audio signal hasa specific loudness that approximates the desired target specificloudness.

By deriving modification parameters from calculations in the loudnessdomain, greater control over perceptual loudness and perceived spectralbalance may be achieved than if such modification parameters werederived in the electrical signal domain. In addition, the use of abasilar-membrane simulating psychoacoustic filterbank or its equivalentin performing loudness domain calculations may provide a more detailedcontrol of the perceived spectrum than in arrangements that derivemodification parameters in the electrical signal domain.

Each of the modifying, deriving, and producing may be dependent on oneor more of a measure of an interfering audio signal, a target specificloudness, an estimate of the specific loudness of the unmodified audiosignal derived from the specific loudness or partial specific loudnessof the modified audio signal, the specific loudness of the unmodifiedaudio signal, and an approximation to the target specific loudnessderived from the specific loudness or partial specific loudness of themodified audio signal.

The modifying or deriving may derive modification parameters at least inpart from one or more of a measure of an interfering audio signal, atarget specific loudness, an estimate of the specific loudness of theunmodified audio signal derived from the specific loudness or partialspecific loudness of the modified audio signal, the specific loudness ofthe unmodified audio signal, and an approximation to the target specificloudness derived from the specific loudness or partial specific loudnessof the modified audio signal.

More particularly, the modifying or deriving may derive modificationparameters at least in part from

(1) one of

-   -   a target specific loudness, and    -   an estimate of the specific loudness of the unmodified audio        signal received from the specific loudness of the modified audio        signal, and

(2) one of

-   -   the specific loudness of the unmodified audio signal, and    -   an approximation to the target specific loudness derived from        the specific loudness of the modified audio signal,        or, when an interfering audio signal is to be taken into        account, the modifying or deriving may derive modification        parameters at least in part from

(1) a measure of an interfering audio signal,

(2) one of

-   -   a target specific loudness, and    -   an estimate of the specific loudness of the unmodified audio        signal derived from the partial specific loudness of the        modified audio signal, and

(3) one of

-   -   the specific loudness of the unmodified audio signal, and    -   an approximation to the target specific loudness derived from        the partial specific loudness of the modified audio signal.

A feed-forward arrangement may be employed in which the specificloudness is derived from the audio signal and wherein the targetspecific loudness is received from a source external to the method orfrom a storing when the modifying or deriving includes storing a targetspecific loudness. Alternatively, a hybrid feed-forward/feedbackarrangement may be employed in which an approximation to the targetspecific loudness is derived from the modified audio signal and whereinthe target specific loudness is received from a source external to themethod or from a storing when the modifying or deriving includes storinga target specific loudness.

The modifying or deriving may include one or more processes forobtaining, explicitly or implicitly, the target specific loudness, oneor ones of which calculates, explicitly or implicitly, a function of theaudio signal or measure of the audio signal. In one alternative, afeed-forward arrangement may be employed in which the specific loudnessand the target specific loudness are derived from the audio signal, thederivation of the target specific loudness employing a function of theaudio signal or measure of the audio signal. In another alternative, ahybrid feed-forward/feedback arrangement may be employed in which anapproximation to the target specific loudness is derived from themodified audio signal and the target specific loudness is derived fromthe audio signal, the derivation of the target specific loudnessemploying a function of the audio signal or measure of the audio signal.

The modifying or deriving may include one or more processes forobtaining, explicitly or implicitly, an estimate of the specificloudness of the unmodified audio signal in response to the modifiedaudio signal, one or ones of which calculates, explicitly or implicitly,the inverse of a function of the audio signal or measure of the audiosignal. In one alternative, a feedback arrangement is employed in whichan estimate of the specific loudness of the unmodified audio signal andan approximation to the target specific loudness are derived from themodified audio signal, the estimate of the specific loudness beingcalculated using the inverse of a function of the audio signal ormeasure of the audio signal. In another alternative, a hybridfeed-forward/feedback arrangement is employed in which the specificloudness is derived from the audio signal and the estimate of thespecific loudness of the unmodified audio signal is derived from themodified audio signal, the derivation of the estimate being calculatedusing the inverse of said function of the audio signal or measure of theaudio signal.

Modification parameters may be applied to the audio signal to produce amodified audio signal.

Another aspect of the invention is that there may be a temporal and/orspatial separation of processes or devices so that there is, in effect,an encoder or encoding and also a decoder or decoding. For example,there may be an encoding/decoding system in which the modifying orderiving may either transmit and receive or store and also reproduce theaudio signal and either (1) modification parameters or (2) a targetspecific loudness or a representation of a target specific loudness.Alternatively, there may be, in effect, only an encoder or encoding inwhich there is either a transmitting or storing of the audio signal and(1) modification parameters or (2) a target specific loudness orrepresentation of target specific loudness. Alternatively, as mentionedabove, there may be, in effect, only a decoder or decoding in whichthere is a reception and reproduction of the audio signal and (1)modification parameters or (2) a target specific loudness orrepresentation of target specific loudness.

An aspect of the invention is directed to a method for adjusting a levelof an audio signal in an audio processing apparatus, where the methodincludes dividing an audio signal into a plurality of frequency bands,and obtaining modification parameters for at least some of the pluralityof frequency bands. The modification parameters may comprise filtercoefficients and amplitude scale factors, each amplitude scale factorrespectively operating in a frequency band of a plurality of frequencybands. In one example, each amplitude scale factor may represent anaverage energy over a frequency band and a time segment. The method mayfurther include deriving gain factors for at least some of the pluralityof frequency bands. The gain factors may be determined at least in partfrom the amplitude scale factors. The gain factors may be smoothed. Thesmoothing of the gain factors may be optional.

A level of noise may be determined from noise compensation factors. Thegain factors may be applied to at least some of the frequency bands togenerate gain adjusted frequency bands. The method may further includeadjusting the level of noise based on the gain adjusted frequency bands.The method may further include filtering at least some of the frequencybands with a filter generated with the filter coefficients andsynthesizing the plurality of frequency bands to generate an outputaudio signal. The gain factors may be both time and frequency varying.

The method may further comprise band smoothing of the amplitude scalefactors. The plurality of frequency bands may be derived from ananalysis filter bank. The adjusting may be performed by a synthesisfilterbank.

The filter coefficients may be applied to time-varying filters. Thefilter coefficients may be time-varying. The amplitude factors may befrequency varying. The method may further include delaying at least oneof the plurality of the frequency bands, in one example delaying themprior to obtaining the modification parameters.

An aspect of the invention may be directed to audio processing apparatusfor adjusting a level of an audio signal. The apparatus may include ananalysis filterbank for dividing an audio signal into a plurality offrequency bands. The apparatus may further include a parameter generatorfor obtaining modification parameters for at least some of the pluralityof frequency bands. The modification parameters may comprise filtercoefficients and amplitude scale factors. Each amplitude scale factormay respectively operate in a frequency band of a plurality of frequencybands. Each amplitude scale factor may represent an average energy overa frequency band and a time segment. The apparatus may further include aprocessor for deriving gain factors for at least some of the pluralityof frequency bands. The gain factors may be determined at least in partfrom the amplitude scale factors. The apparatus may further include asmoother for smoothing the gain factors. The apparatus may furtherinclude a processor for determining a level of noise from noisecompensation factors. The apparatus may further include a first adjusterfor applying the gain factors to at least some of the frequency bands togenerate gain adjusted frequency bands. The apparatus may furtherinclude a second adjuster for adjusting the level of noise based on thegain adjusted frequency bands. The apparatus may further include afilter for filtering at least some of the frequency bands. The filtermay be generated based on the filter coefficients. The apparatus mayfurther include a synthesis filterbank for synthesizing the plurality offrequency bands to generate an output audio signal. The gain factors maybe both time and frequency varying.

The apparatus may further comprise a band smoother for smoothing theamplitude scale factors. The filter coefficients may be applied totime-varying filters. The filter coefficients may be time-varying. Theamplitude factors may be frequency varying. The method may furtherinclude delaying at least one of the plurality of the frequency bands,in one example delaying them prior to obtaining the modificationparameters.

An aspect of the invention may be directed to a non-transitory computerreadable medium, storing software instructions for controlling aperceptual loudness of a digital audio signal, which when executed byone or more processors cause performance of the steps of the methodoutlined above and in claim 1.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram illustrating an example of afeed-forward implementation according to aspects of the invention.

FIG. 2 is a functional block diagram illustrating an example of afeedback implementation according to aspects of the invention.

FIG. 3 is a functional block diagram illustrating an example of a hybridfeed-forward/feedback implementation according to aspects of theinvention.

FIG. 4 is a functional block diagram illustrating an example of anotherhybrid feed-forward/feedback implementation according to aspects of theinvention

FIG. 5 is a functional block diagram illustrating the manner in whichthe unmodified audio signal and the modification parameters asdetermined by any one of the feed-forward, feedback, and hybridfeed-forward feedback arrangements may be stored or transmitted for use,for example, in a temporally or spatially separated device or process.

FIG. 6 is a functional block diagram illustrating the manner in whichthe unmodified audio signal and a target specific loudness orrepresentation thereof as determined by any one of the feed-forward,feedback, and hybrid feed-forward feedback arrangements may be stored ortransmitted for use, for example, in a temporally or spatially separateddevice or process.

FIG. 7 is a schematic functional block diagram or schematic flow chartshowing an overview of an aspect of the present invention.

FIG. 8 is an idealized characteristic response of a linear filter P(z)suitable as a transmission filter in an embodiment of the presentinvention in which the vertical axis is attenuation in decibels (dB) andthe horizontal axis is a logarithmic base 10 frequency in Hertz (Hz).

FIG. 9 shows the relationship between the ERB frequency scale (verticalaxis) and frequency in Hertz (horizontal axis).

FIG. 10 shows a set idealized auditory filter characteristic responsesthat approximate critical banding on the ERB scale. The horizontal scaleis frequency in Hertz and the vertical scale is level in decibels.

FIG. 11 shows the equal loudness contours of ISO 226. The horizontalscale is frequency in Hertz (logarithmic base 10 scale) and the verticalscale is sound pressure level in decibels.

FIG. 12 shows the equal loudness contours of ISO 226 normalized by thetransmission filter P(z). The horizontal scale is frequency in Hertz(logarithmic base 10 scale) and the vertical scale is sound pressurelevel in decibels.

FIG. 13a is an idealized chart showing wideband and multiband gains forloudness scaling of 0.25 on a segment of female speech. The horizontalscale is ERB bands and the vertical scale is relative gain in decibels(dB).

FIG. 13b is an idealized chart showing the specific loudness,respectively, of an original signal, a wideband gain-modified signal,and a multiband gain-modified signal. The horizontal scale is ERB bandsand the vertical scale is specific loudness (sone/ERB).

FIG. 14a is an idealized chart showing: L_(o)[t] as a function ofL_(i)[t] for typical AGC. The horizontal scale is log (L_(i)[t]) and thevertical scale is log(L_(o)[t]).

FIG. 14b is an idealized chart showing: L_(o)[t] as a function ofL_(i)[t] for typical DRC. The horizontal scale is log (L_(i)[t]) and thevertical scale is log (L_(o)[t]).

FIG. 15 is an idealized chart showing a typical band-smoothing functionfor multiband DRC. The horizontal scale is band number and the verticalscale is the gain output for the band b.

FIG. 16 is a schematic functional block diagram or schematic flow chartshowing an overview of an aspect of the present invention.

FIG. 17 is a schematic functional block diagram or schematic flow chartsimilar to FIG. 1 that also includes compensation for noise in aplayback environment.

BEST MODE FOR CARRYING OUT THE INVENTION

FIGS. 1 through 4 show functional block diagrams illustrating possiblefeed-forward, feedback, and two versions of hybrid feed-forward/feedbackimplementation examples according to aspects of the invention.

Referring to the example of a feed-forward topology in FIG. 1, an audiosignal is applied to two paths: (1) a signal path having a process ordevice 2 (“Modify Audio Signal”) capable of modifying the audio inresponse to modification parameters, and (2) a control path having aprocess or device 4 (“Generate Modification Parameters”) capable ofgenerating such modification parameters. The Modify Audio Signal 2 inthe FIG. 1 feed-forward topology example and in each of the FIGS. 2-4examples may be a device or process that modifies the audio signal, forexample, its amplitude, in a frequency- and/or time-varying manner inaccordance with modification parameters M received from the GenerateModification Parameters 4 (or from counterpart processes or devices 4′,4″ and 4′″ in each of the FIGS. 2-4 examples, respectively). TheGenerate Modification Parameters 4 and its counterparts in FIGS. 2-4each operate at least partly in the perceptual loudness domain. TheModify Audio Signal 2 operates in the electrical signal domain andproduces a modified audio signal in each of the FIG. 1-4 examples. Alsoin each of the FIG. 1-4 examples, the Modify Audio Signal 2 and theGenerate Modification Parameters 4 (or its counterparts) modify theaudio signal to reduce the difference between its specific loudness anda target specific loudness.

In the FIG. 1 feed-forward example, process or device 4 may includeseveral processes and/or devices: a “Calculate Target Specific Loudness”process or device 6 that calculates a target specific loudness inresponse to the audio signal or a measure of the audio signal such asthe specific loudness of the audio signal, a “Calculate SpecificLoudness” process or device 8 that calculates the specific loudness ofthe audio signal in response to the audio signal or a measure of theaudio signals such as its excitation, and a “Calculate ModificationParameters” process or device 10 that calculates the modificationparameters in response to the specific loudness and the target specificloudness. The Calculate Target Specific Loudness 6 may perform one ormore functions “F”, each of which may have function parameters. Forexample, it may calculate the specific loudness of the audio signal andthen apply one or more functions F to it to provide a target specificloudness. This is indicated schematically in FIG. 1 as a “SelectFunction(s) F and Function(s) Parameter(s)” input to process or device6. Instead of being calculated by device or process 6, the targetspecific loudness may be provided by a storing process or device (shownschematically as a “Stored” input to process or device 10) included inor associated with the Generate Modification Parameters 4, or by asource external to the overall process or device (shown schematically asthe “External” input to process or device 10). Thus, the modificationparameters are based at least in part on calculations in the perceptual(psychoacoustic) loudness domain (i.e., at least the specific loudnessand, in some cases, the target specific loudness calculations).

The calculations performed by processes or devices 6, 8 and 10 (and byprocesses or devices 12, 14, 10′ in the FIG. 2 example, 6, 14, 10″ inthe FIG. 3 example, and 8, 12, 10′″ in the FIG. 4 example) may beperformed explicitly and/or implicitly. Examples of implicit performanceinclude (1) a lookup table whose entries are based in whole or in parton specific loudness and/or target specific loudness and/or modificationparameter calculations, and (2) a closed-form mathematical expressionthat is inherently based in whole or in part on specific loudness and/ortarget specific loudness and/or modification parameters.

Although the calculation processes or devices 6, 8 and 10 of the FIG. 1example (and the processes or devices 12, 14, 10′ in the FIG. 2 example,6, 14, 10″ in the FIG. 3 example, and 8, 12, 10′″ in the FIG. 4 example)are shown schematically and described as separate, this is for purposesof explanation only. It will be understood that ones or all of theseprocesses or devices may be combined in a single process or device orcombined variously in multiple processes or devices. For example, in thearrangement of FIG. 9 below, a feed-forward topology as in the exampleof FIG. 1, the process or device that calculates modification parametersdoes so in response to the smoothed excitation derived from the audiosignal and a target specific loudness. In the FIG. 9 example, the deviceor process that calculates modification parameters implicitly calculatesspecific loudness of the audio signal.

As an aspect of the present invention, in the example of FIG. 1 and inother examples of embodiments of the invention herein, the targetspecific loudness ({circumflex over (N)}[b,t]) may be calculated byscaling the specific loudness (N[b,t]) with one or more scaling factors.The scaling may be a time- and frequency-varying scale factor Ξ[b,t]scaling of the specific loudness as in the relationship{circumflex over (N)}[b,t]=Ξ[b,t]N[b,t],a time-varying, frequency-invariant scale factor Φ[t] scaling of thespecific loudness as in the relationship{circumflex over (N)}[b,t]=Φ[t]N[b,t],a time-invariant, frequency-varying scale factor Θ[b] scaling of thespecific loudness as in the relationship{circumflex over (N)}[b,t]=Θ[b]N[b,t],ora scale factor α scaling of the specific loudness of the audio signal asin the relationship{circumflex over (N)}[b,t]=αN[b,t],where b is a measure of frequency (e.g., the band number) and t is ameasure of time (e.g., the block number). Multiple scalings may also beemployed, using multiple instances of a particular scaling and/orcombinations of particular scalings. Examples of such multiple scalingsare given below. In some cases, as explained further below, the scalingmay be a function of the audio signal or measure of the audio signal. Inother cases, also as explained further below, when the scaling is not afunction of a measure of the audio signal, the scaling may be otherwisedetermined or supplied. For example, a user could select or apply atime- and frequency-invariant scale factor α or a time-invariant,frequency-varying scale factor Θ[b] scaling.

Thus, the target specific loudness may be expressed as one or morefunctions F of the audio signal or measure of the audio signal (thespecific loudness being one possible measure of the audio signal):{circumflex over (N)}[b,t]=F(N[b,t]).

Provided that the function or functions F is invertible, the specificloudness (N[b,t] of the unmodified audio signal may be calculated as theinverse function or functions F⁻¹ of the target specific loudness({circumflex over (N)}[b,t]):N[b,t]=F ⁻¹({circumflex over (N)}[b,t]).

As will be seen below, the inverse function or functions F⁻¹ iscalculated in the feedback and hybrid feed-forward/feedback examples ofFIGS. 2 and 4.

A “Select Function(s) and Function Parameter(s)” input for CalculateTarget Specific Loudness 6 is shown to indicate that the device orprocess 6 may calculate the target specific loudness by applying one ormore functions in accordance with one or more function parameters. Forexample, the Calculate Target Specific Loudness 8 may calculate thefunction or functions “F” of the specific loudness of the audio signalin order to define the target specific loudness. For example, the“Select Function(s) and Function Parameter(s)” input may select one ormore particular functions that fall into one or more of the above typesof scaling, along with one or more function parameters, such asconstants (e.g., scale factors) pertaining to the functions.

The scaling factors associated with a scaling may serve as arepresentation of the target specific loudness inasmuch as the targetspecific loudness may be computed as a scaling of the specific loudness,as indicated above. Thus, in the FIG. 9 example, described below andmentioned above, the lookup table may be indexed by scale factors andexcitations, such that the calculation of specific loudness and targetspecific loudness are inherent in the table.

Whether employing a lookup table, a closed-form mathematical expression,or some other technique, the operation of the Generate ModificationParameters 4 (and its counterpart processes or devices 4′, 4″ and 4′″ ineach of the FIGS. 2-4 examples) is such that the calculations are basedin the perceptual (psychoacoustic) loudness domain even though specificloudness and target specific loudness may not be explicitly calculated.Either there is an explicit specific loudness or there is a notional,implicit specific loudness. Similarly, either there is an explicittarget specific loudness or there is a notional, implicit targetspecific loudness. In any case, the calculation of modificationparameters seeks to generate modification parameters that modify theaudio signal to reduce the difference between specific loudness and atarget specific loudness.

In a playback environment having a secondary interfering audio signal,such as noise, the Calculate Modification Parameters 10 (and itscounterpart processes or devices 10′, 10″ and 10′″ in each of the FIGS.2-4 examples, respectively) may also receive as an optional input ameasure of such a secondary interfering audio signal or the secondaryinterfering signal itself as one of its inputs. Such an optional inputis shown in FIG. 1 (and in FIGS. 2-4) with a dashed lead line. Themeasure of a secondary interfering signal may be its excitation such asin the example of FIG. 17, described below. The application of a measureof the interfering signal or the signal itself (it being assumed thatthe interfering signal is separately available for processing) to theCalculate Modification Parameters process or devices 10 in FIG. 1 (andits counterpart processes or devices 10′, 10″ and 10′″ in each of theFIGS. 2-4 examples, respectively) permits a suitably configured suchprocess or device to calculate modification parameters that take theinterfering signal into account as explained further below under theheading “Noise Compensation.” In the examples of FIGS. 2-4, thecalculation of partial specific loudness assumes that a suitable measureof an interfering signal is applied not only to the respective CalculateModification Parameters 10′, 10″, or 10′″, but also to a “CalculateApproximation of Specific Loudness of Unmodified Audio” process ordevice 12 and/or a “Calculate Approximation of Target Specific Loudness”process or device 14 in order to facilitate the calculation of partialspecific loudness by that function or device. In the FIG. 1 feed-forwardexample, partial specific loudness is not explicitly calculated—theCalculate Modification Parameters 10 of FIG. 1 calculates theappropriate modification parameters to make the partial specificloudness of the modified audio approximate the target specific loudness.This is explained further below under the heading “Noise Compensation.”mentioned above.

As mentioned above, in each of the FIG. 1-4 examples, the modificationparameters M, when applied to the audio signal by the Audio SignalModifier 2, reduce the difference between the specific loudness or thepartial specific loudness of the resulting modified audio and the targetspecific loudness. Ideally, the specific loudness of the modified audiosignal closely approximates or is the same as the target specificloudness. The modification parameters M may, for example, take the formof time-varying gain factors applied to the frequency bands derived froma filterbank or to the coefficients of a time-varying filter.Accordingly, in all of the FIG. 1-4 examples, Modify Audio Signal 2 maybe implemented as, for example, a plurality of amplitude scalers, eachoperating in a frequency band, or a time-varying filter (e.g., amultitapped FIR filter or a multipole IIR filter).

Here and elsewhere in this document, the use of the same referencenumeral indicates that the device or process may be substantiallyidentical to another or others bearing the same reference numeral.Reference numerals bearing prime numbers (e.g., “10′”) indicates thatthe device or process is similar to in structure or function but may bea modification of another or others bearing the same basic referencenumeral or primed versions thereof.

Under certain constraints, a nearly equivalent feedback arrangement ofthe feed-forward example of FIG. 1 may be realized. FIG. 2 depicts suchan example in which the audio signal is also applied to a Modify AudioSignal process or device 2 in a signal path. The process or device 2also receives the modification parameters M from a control path in whicha Generate Modification Parameters process or device 4′ in a feedbackarrangement receives as its input the modified audio signal from theoutput of the Modify Audio Signal 2. Thus, in the FIG. 2 example, themodified audio rather than the unmodified audio is applied to a controlpath. The Modify Audio Signal process or device 2 and the GenerateModification Parameters process or device 4′ modify the audio signal toreduce the difference between its specific loudness and a targetspecific loudness. The process or device 4′ may include severalfunctions and or devices: a “Calculate Approximation of SpecificLoudness of Unmodified Audio” process or device 12, a “CalculateApproximation of Target Specific Loudness” process or device 14, and a“Calculate Modification Parameters” process or device 10′ thatcalculates the modification parameters.

With the constraint that the function or functions F is invertible, theprocess or device 12 estimates the specific loudness of the unmodifiedaudio signal by applying the inverse function F⁻¹ to the specificloudness or partial specific loudness of the modified audio signal. Thedevice or process 12 may calculate an inverse function F⁻¹, as describedabove. This is indicated schematically in FIG. 2 as a “Select InverseFunction(s) F⁻¹ and Function(s) Parameter(s)” input to process or device12. The “Calculate Approximation of Target Specific Loudness” 14operates by calculating the specific loudness or partial specificloudness of the modified audio signal. Such specific loudness or partialspecific loudness is an approximation of the target specific loudness.The approximation of the specific loudness of the unmodified audiosignal and the approximation of the target specific loudness are used bythe Calculate Modification Parameters 10′ to derive modificationparameters M, which, if applied to the audio signal by the Modify AudioSignal 2, reduce the difference between the specific loudness or thepartial specific loudness of the modified audio signal and the targetspecific loudness. As mentioned above, these modification parameters Mmay, for example, take the form of time-varying gains applied to thefrequency bands of a filterbank or the coefficients of a time-varyingfilter. In Calculate Modification Parameters 10″ practical embodimentsthe feedback loop may introduce a delay between the computation andapplication of the modification parameters M.

As mentioned above, in a playback environment having a secondaryinterfering audio signal, such as noise, the Calculate ModificationParameters 10,′ the Calculate Approximation of Specific Loudness ofUnmodified Audio 12, and the Calculate Approximation of Target SpecificLoudness 14 may each also receive as an optional input a measure of sucha secondary interfering audio signal or the secondary interfering signalitself as one of its inputs and process or device 12 and process ordevice 14 may each calculate the partial specific loudness of themodified audio signal. Such optional inputs are shown in FIG. 2 usingdashed lead lines.

As mentioned above, hybrid feed-forward/feedback implementation examplesof aspects of the invention are possible. FIGS. 3 and 4 show twoexamples of such implementations. In the FIGS. 3 and 4 examples, as inthe FIGS. 1 and 2 example, the audio signal is also applied to a ModifyAudio Signal process or device 2 in a signal path, but GenerateModification Parameters (4″ in FIGS. 3 and 4′″ in FIG. 4) in respectivecontrol paths each receive both the unmodified audio signal and themodified audio signal. In both the FIGS. 3 and 4 examples, the ModifyAudio Signal 2 and Generate Modification Parameters (4″ and 4′″,respectively) modify the audio signal to reduce the difference betweenits specific loudness, which may be implicit, and a target specificloudness, which may also be implicit.

In the FIG. 3 example, the Generate Modification Parameters process ordevice 4′ may include several functions and or devices: a CalculateTarget Specific Loudness 6 as in the FIG. 1 example, a CalculateApproximation of Target Specific Loudness 14, as in the FIG. 2 feedbackexample, and a “Calculate Modification Parameters” process or device10″. As in the FIG. 1 example, in the feed-forward portion of thishybrid feed-forward/feedback example, the Calculate Target SpecificLoudness 6 may perform one or more functions “F”, each of which may havefunction parameters. This is indicated schematically in FIG. 3 as a“Select Function(s) F and Function(s) Parameter(s)” input to process ordevice 6. In the feedback portion of this hybrid feed-forward/feedbackexample, the modified audio signal is applied to a CalculateApproximation of Target Specific Loudness 14, as in the FIG. 2 feedbackexample. Process or device 14 operates in the FIG. 3 example as it doesin the FIG. 2 example by calculating the specific loudness or partialspecific loudness of the modified audio signal. Such specific loudnessor partial specific loudness is an approximation of the target specificloudness. The target specific loudness (from process or device 6) andthe approximation of the target specific loudness (from process ordevice 14) are applied to the Calculate Modification Parameters 10″ toderive modification parameters M, which, if applied to the audio signalby the Modify Audio Signal 2, reduce the difference between the specificloudness of the unmodified audio signal and the target specificloudness. As mentioned above, these modification parameters M may, forexample, take the form of time-varying gains applied to the frequencybands of a filterbank or the coefficients of a time-varying filter. Inpractical embodiments, the feedback loop may introduce a delay betweenthe computation and application of the modification parameters M. Asmentioned above, in a playback environment having a secondaryinterfering audio signal, such as noise, the Calculate ModificationParameters 10″ and the Calculate Approximation of Target SpecificLoudness 14 may each also receive as an optional input a measure of sucha secondary interfering audio signal or the secondary interfering signalitself as one of its inputs and process or device 14 may calculate thepartial specific loudness of the modified audio signal. The optionalinputs are shown in FIG. 3 using dashed lead lines.

The Calculate Modification Parameters 10″ may employ an error detectingdevice or function, such that differences between its target specificloudness and target specification loudness approximation inputs adjustthe Modification Parameters so as to reduce the differences between theapproximation of the target specific loudness and the “actual” targetspecific loudness. Such adjustments reduce the differences between thespecific loudness of the unmodified audio signal, and the targetspecific loudness, which may be implicit. Thus, the modificationparameters M may be updated based on an error between the targetspecific loudness, computed in the feed-forward path from the specificloudness of the original audio using the function F, and the targetspecific loudness approximation computed in the feedback path fromspecific loudness or partial specific loudness of the modified audio.

In the FIG. 4 example, an alternative feed-forward/feedback example isshown. This alternative differs from the example of FIG. 3 in that theinverse function or functions F⁻¹ is calculated in the feedback pathrather than the function or functions F being calculated in thefeed-forward path. In the FIG. 4 example, the Generate ModificationParameters process or device 4′ may include several functions and ordevices: a Calculate Specific Loudness 8, as in the FIG. 1 feed-forwardexample, Calculate Approximation of Specific Loudness of UnmodifiedAudio 12, as in the FIG. 2 feedback example, and a CalculateModification Parameters 10′″. The Calculate Specific Loudness 8, as inthe FIG. 1 feed-forward example, provides, as an input to the CalculateModification Parameters 10′″, the specific loudness of the unmodifiedaudio signal. As in the FIG. 2 feedback example, with the constraintthat the function or functions F is invertible, the process or device 12estimates the specific loudness of the unmodified audio signal byapplying the inverse function F⁻¹ to the specific loudness or partialspecific loudness of the modified audio signal. A “Select InverseFunction(s) and Inverse Function(s) Parameter(s)” input for CalculateApproximation of Specific Loudness of Unmodified Audio 12 is shown toindicate that the device or process 12 may calculate an inverse functionF⁻¹, as described above. This is indicated schematically in FIG. 4 as a“Select Inverse Function(s) F⁻¹ and Function(s) Parameter(s)” input toprocess or device 12. Thus, process or device 12 provides as anotherinput to the Calculate Modification Parameters 10′″ an approximation tothe specific loudness of the unmodified audio signal.

As in the examples of FIGS. 1-3, the Calculate Modification Parameters10′″ derives modification parameters M, which, if applied to the audiosignal by the Modify Audio Signal 2, reduce the difference between thespecific loudness of the unmodified audio signal and the target specificloudness, which is implicit in this example. As mentioned above, themodification parameters M may, for example, take the form oftime-varying gains applied to the frequency bands of a filterbank or thecoefficients of a time-varying filter. In practical embodiments, thefeedback loop may introduce a delay between the computation andapplication of the modification parameters M. As mentioned above, in aplayback environment having a secondary interfering audio signal, suchas noise, the Calculate Modification Parameters 10′″ and the CalculateApproximation of Specific Loudness of the Unmodified Audio 12 may eachalso receive as an optional input a measure of such a secondaryinterfering audio signal or the secondary interfering signal itself asone of its inputs and process or device 12 may calculate the partialspecific loudness of the modified audio signal. The optional inputs areshown in FIG. 4 using dashed lead lines.

The Calculate Modification Parameters 10′″ may employ an error detectingdevice or function, such that differences between its specific loudnessand specific loudness approximation inputs produce outputs that adjustthe Modification Parameters so as to reduce the differences between theapproximation of the specific loudness and the “actual” specificloudness. Because the approximation of the specific loudness is derivedfrom the specific loudness or partial specific loudness of the modifiedaudio, which can be viewed as an approximation of the target specificloudness, such adjustments reduce the differences between the specificloudness of the modified audio signal and the target specific loudness,which is inherent in the function or functions F⁻¹. Thus, themodification parameters M may be updated based on an error between thespecific loudness, computed in the feed-forward path from the originalaudio, and the specific loudness approximation computed, using theinverse function or functions F⁻¹, in the feedback path from specificloudness or partial specific loudness of the modified audio. Due to thefeedback path, practical implementations may introduce a delay betweenthe update and application of the modification parameters.

Although the modification parameters M in the examples of FIGS. 1-4 whenapplied to a Modify Audio Signal process or device 2 reduce thedifference between the specific loudness of the audio signal and thetarget specific loudness, in practical embodiments the correspondingmodification parameters produced in response to the same audio signalmay not be identical to each other.

Although not critical or essential to aspects of the present invention,calculation of the specific loudness of the audio signal or the modifiedaudio signal may advantageously employ techniques set forth in saidInternational Patent Application No. PCT/US2004/016964, published as WO2004/111964 A2, wherein the calculating selects, from a group of two ormore specific loudness model functions, one or a combination of two ormore of the specific loudness model functions, the selection of which iscontrolled by the measure of characteristics of the input audio signal.The description of Specific Loudness 104 of FIG. 1, below, describessuch an arrangement.

In accordance with further aspects of the invention, the unmodifiedaudio signal and either (1) the modification parameters or (2) thetarget specific loudness or a representation of the target specificloudness (e.g., scale factors usable in calculating, explicitly orimplicitly, target specific loudness) may be stored or transmitted foruse, for example, in a temporally and/or spatially separated device orprocess. The modification parameters, target specific loudness, orrepresentation of the target specific loudness may be determined in anysuitable way, as, for example, in one of the feed-forward, feedback, andhybrid feed-forward feedback arrangement examples of FIGS. 1-4, asdescribed above. In practice, a feed-forward arrangement, such as in theexample of FIG. 1, is the least complex and fastest inasmuch as itavoids calculations based on the modified audio signal. An example oftransmitting or storing the unmodified audio and the modificationparameters is shown in FIG. 5, while an example of the transmitting orstoring the unmodified audio and the target specific loudness or arepresentation of the target specific loudness is shown in FIG. 6.

An arrangement such as in the example of FIG. 5 may be used totemporally and/or spatially separate the application of the modificationparameters to the audio signal from the generation of such modificationparameters. An arrangement such as in the example of FIG. 6 may be usedto temporally and/or spatially separate both the generation andapplication of the modification parameters from the generation of thetarget specific loudness or representation of it. Both types ofarrangements make possible a simple low-cost playback or receptionarrangement that avoids the complexity of generating the modificationparameters or of generating the target specific loudness. Although aFIG. 5 type arrangement is simpler than a FIG. 6 type arrangement, theFIG. 6 arrangement has the advantage that the information required to bestored or transmitted may be much less, particularly when arepresentation of the target specific loudness, such as one or morescale factors are stored or transmitted. Such a reduction in informationstorage or transmission may be particularly useful in low-bit-rate audioenvironments.

Accordingly, further aspects of the present invention are the provisionof a device or process (1) that receives or plays back, from a store ortransmit device or process, modification parameters M and applies themto an audio signal that is also received or (2) that receives or playsback, from a store or transmit device or process, a target specificloudness or representation of a target specific loudness, generatesmodification parameters M by applying the target specific loudness orrepresentation thereof to the audio signal that is also received (or toa measure of the audio signal such as its specific loudness, which maybe derived from the audio signal), and applies the modificationparameters M to the received audio signal. Such devices or processes maybe characterized as decoding processes or decoders; while the devices orprocesses required to produce the stored or transmitted information maybe characterized as encoding processes or encoders. Such encodingprocesses or encoders are those portions of the FIGS. 1-4 arrangementexamples that are usable to produce the information required by therespective decoding processes or decoders. Such decoding processors ordecoders may be associated or operative with virtually any type ofprocess or device that processes and/or reproduces sound.

In one aspect of the invention, as in the example of FIG. 5, theunmodified audio signal and the modification parameters M produced by,for example, a modification parameter generating process or generatorsuch as Generate Modification Parameters 4 of FIG. 1, 4′ of FIG. 2, 4″of FIG. 3 or 4′″ of FIG. 4 may be applied to any suitable storage ortransmission device or function (“Store or Transmit”) 16. In the case ofusing the feed-forward example of FIG. 1 as an encoding process or anencoder, the Modify Audio Signal 2 would not be required to generate themodified audio and could be omitted if there is no requirement toprovide the modified audio at the temporal or spatial location of theencoder or encoding process. The Store or Transmit 16 may include, forexample, any suitable magnetic, optical or solid-state storage andplayback devices or any suitable wired or wireless transmission andreception devices, the choice thereof not being critical to theinvention. The played-back or received modification parameters may thenbe applied to a Modify Audio Signal 2, of the type employed in theexamples of FIGS. 1-4, in order to modify the played-back or receivedaudio signal so that its specific loudness approximates the targetspecific loudness of or inherent in the arrangement in which themodification parameters were derived. The modification parameters may bestored or transmitted in any of various ways. For example, they may bestored or transmitted as metadata accompanying the audio signal, theymay be sent in separate paths or channels, they may besteganographically encoded in the audio, they may be multiplexed, etc.The use of the modification parameters to modify the audio signal may beoptional and, if optional, their use may be selectable, for example, bya user. For example, the modification parameters if applied to the audiosignal might reduce the dynamic range of the audio signal. Whether ornot to employ such dynamic range reduction could be selectable by auser.

In another aspect of the invention, as in the example of FIG. 6, theunmodified audio signal and the target specific loudness orrepresentation of the target specific loudness may be applied to anysuitable storage or transmission device or function (“Store orTransmit”) 16. In the case of using a feed-forward configuration, suchas the example of FIG. 1, as an encoding process or an encoder, neithera Calculate Modification Parameters 10 type process or device nor aModify Audio Signal 2 type process or device would be required and couldbe omitted if there is no requirement to provide either the modificationparameters or the modified audio at the temporal or spatial location ofthe encoder or encoding process. As in the case of the FIG. 5 example,the Store or Transmit 16 may include, for example, any suitablemagnetic, optical or solid-state storage and playback devices or anysuitable wired or wireless transmission and reception devices, thechoice thereof not being critical to the invention. The played-back orreceived target specific loudness or representation of the targetspecific loudness may then be applied, along with the unmodified audio,to a Calculate Modification Parameters 10, of the type employed in theexample of FIG. 1, or to a Calculate Modification Parameters 10″, of thetype employed in the example of FIG. 3, in order to provide modificationparameters M that may then be applied to Modify Audio Signal 2, of thetype employed in the examples of FIGS. 1-4, in order to modify theplayed-back or received audio signal so that its specific loudnessapproximates the target specific loudness of or inherent in thearrangement in which the modification parameters were derived. Althoughthe target specific loudness or representation thereof may be mostreadily obtained in an encoding process or encoder of the FIG. 1 exampletype, the target specific loudness or representation thereof or anapproximation to the target specific loudness or representation thereofmay be obtained in an encoding process or encoder of the FIG. 2 through4 example types (approximations are calculated in processes or devices14 of FIGS. 2 and 3 and in process or device 12 of FIG. 4). The targetspecific loudness or representation thereof may be stored or transmittedin any of various ways. For example, it may be stored or transmitted asmetadata accompanying the audio signal, it may be sent in separate pathsor channels, it may be steganographically encoded in the audio, it maybe multiplexed, etc. The use of the modification parameters derived fromthe stored or transmitted target specific loudness or representation tomodify the audio signal may be optional and, if optional, their use maybe selectable, for example, by a user. For example, the modificationparameters if applied to the audio signal might reduce the dynamic rangeof the audio signal. Whether or not to employ such dynamic rangereduction could be selectable by a user.

When implementing the disclosed invention as a digital system, afeed-forward configuration is the most practical, and examples of suchconfigurations are therefore described below in detail, it beingunderstood that the scope of the invention is not so limited.

Throughout this document, terms such as “filter” or “filterbank” areused herein to include essentially any form of recursive andnon-recursive filtering such as IIR filters or transforms, and“filtered” information is the result of applying such filters.Embodiments described below employ filterbanks implemented bytransforms.

FIG. 7 depicts greater details of an exemplary embodiment of an aspectof the invention embodied in a feed-forward arrangement. Audio firstpasses through an analysis filterbank function or device (“AnalysisFilterbank”) 100, which splits the audio signal into a plurality offrequency bands (hence, FIG. 5 shows multiple outputs from AnalysisFilterbank 100, each output representing a frequency band, which outputcarries through the various functions or devices up to an synthesisfilterbank, which sums the bands to a combined wideband signal, asdescribed further below). The response of the filter associated witheach frequency band in the Analysis Filterbank 100 is designed tosimulate the response at a particular location of the basilar membranein the inner ear. The output of each filter in the Analysis Filterbank100 next passes into a transmission filter or transmission filterfunction (“Transmission Filter”) 101 that simulates the filtering effectof the transmission of audio through the outer and middle ear. If onlythe loudness of the audio were to be measured, the transmission filtercould be applied prior to the analysis filterbank, but because theanalysis filterbank outputs are used to synthesize the modified audio itis advantageous to apply the transmission filter after the filterbank.The outputs of Transmission Filter 101 next pass into an excitationfunction or device (“Excitation”) 102, the outputs of which simulate thedistribution of energy along the basilar membrane. The excitation energyvalues may be smoothed across time by a smoothing function or device(“Smoothing”) 103. The time constants of the smoothing function are setin accordance with the requirements of a desired application. Thesmoothed excitation signals are subsequently converted into specificloudness in specific loudness function or device (“Specific Loudness(SL)”) 104. Specific loudness is represented in units of sone per unitfrequency. The specific loudness component associated with each band ispassed into specific loudness modification function or device (“SLModification”) 105. SL Modification 105 takes as its input the originalspecific loudness and then outputs a desired or “target” specificloudness, which, according to an aspect of the present invention, ispreferably a function of the original specific loudness (see the nextheading below, entitled “Target Specific Loudness”). The SL Modification105 may operate independently on each band, or there may exist aninterdependence between or among bands (a frequency smoothing assuggested by the cross-connecting lines in FIG. 7), depending on thedesired effect. Taking as its inputs the smoothed excitation frequencyband components from Excitation 102 and the target specific loudnessfrom the SL Modification 105, a gain solver function or device (“GainSolver”) 106 determines the gain that needs to be applied to each bandof the output of the Analysis Filterbank 100 in order to transform themeasured specific loudness into the target specific loudness. The GainSolver may be implemented in various ways. For example, the Gain Solvermay include an iterative process such as in the manner of that disclosedin said International Patent Application No. PCT/US2004/016964,published as WO 2004/111964 A2, or, alternatively, a table lookup.Although the gains per band generated by the Gain Solver 106 may besmoothed further over time by optional smoothing function or device(“Smoothing”) 107 in order to minimize perceptual artifacts, it ispreferred that temporal smoothing be applied elsewhere in the overallprocess or device, as described elsewhere. Finally, the gains areapplied to respective bands of the Analysis Filterbank 100 through arespective multiplicative combining function or combiner 108, and theprocessed or “modified” audio is synthesized from the gain-modifiedbands in a synthesis filterbank function or device (“SynthesisFilterbank) 110. In addition, the outputs from the analysis filterbankmay be delayed by a delay function or device (“Delay”) 109 prior toapplication of the gains in order to compensate for any latencyassociated with the gain computation. Alternatively, instead ofcalculating gains for use in applying gain modifications in frequencybands, the Gain Solvers 106 may calculate filter coefficients thatcontrol a time-varying filter, such as a multitapped FIR filter or amultipole IIR filter. For simplicity in exposition, aspects of theinvention are mainly described as employing gain factors applied tofrequency bands, it being understood that filter coefficients andtime-varying filters may also be employed in practical embodiments.

In practical embodiments, processing of the audio may be performed inthe digital domain. Accordingly, the audio input signal is denoted bythe discrete time sequence x[n] which has been sampled from the audiosource at some sampling frequency f_(s). It is assumed that the sequencex[n] has been appropriately scaled so that the rms power of x[n] indecibels given by

${RMS}_{dB} = {10{\log_{10}\left( {\frac{1}{L}{\sum\limits_{n = 0}^{L}{x^{2}\lbrack n\rbrack}}} \right)}}$is equal to the sound pressure level in dB at which the audio is beingauditioned by a human listener. In addition, the audio signal is assumedto be monophonic for simplicity of exposition.

Analysis Filterbank 100, Transmission Filter 101, Excitation 102,Specific Loudness 104, Specific Loudness Modification 105, Gain Solver106, and Synthesis Filterbank 110 may be described in greater detail asfollows.

Analysis Filterbank 100

The audio input signal is applied to an analysis filterbank orfilterbank function (“Analysis Filterbank”) 100. Each filter in AnalysisFilterbank 100 is designed to simulate the frequency response at aparticular location along the basilar membrane in the inner ear. TheFilterbank 100 may include a set of linear filters whose bandwidth andspacing are constant on the Equivalent Rectangular Bandwidth (ERB)frequency scale, as defined by Moore, Glasberg and Baer (B. C. J. Moore,B. Glasberg, T. Baer, “A Model for the Prediction of Thresholds,Loudness, and Partial Loudness,” supra).

Although the ERB frequency scale more closely matches human perceptionand shows improved performance in producing objective loudnessmeasurements that match subjective loudness results, the Bark frequencyscale may be employed with reduced performance.

For a center frequency fin hertz, the width of one ERB band in hertz maybe approximated as:ERB(f)=24.7(4.37f/1000+1)  (1)

From this relation a warped frequency scale is defined such that at anypoint along the warped scale, the corresponding ERB in units of thewarped scale is equal to one. The function for converting from linearfrequency in hertz to this ERB frequency scale is obtained byintegrating the reciprocal of Eqn. 1:

$\begin{matrix}{{{HzToERB}(f)} = {{\int{\frac{1}{24.7\left( {{4.37{f/1000}} + 1} \right)}{df}}} = {21.4{\log_{10}\left( {{4.37{f/1000}} + 1} \right)}}}} & \left( {2a} \right)\end{matrix}$

It is also useful to express the transformation from the ERB scale backto the linear frequency scale by solving Eqn. 2a for f:

$\begin{matrix}{{{{ERBToHz}(e)} = {f = {\frac{1000}{4.37}10^{({{e/21.4} - 1})}}}},} & \left( {2b} \right)\end{matrix}$

where e is in units of the ERB scale. FIG. 9 shows the relationshipbetween the ERB scale and frequency in hertz.

The Analysis Filterbank 100 may include B auditory filters, referred toas bands, at center frequencies f_(c)[1] . . . f_(c)[B] spaced uniformlyalong the ERB scale. More specifically,f _(c)[1]=f _(min)  (3a)f _(c)[b]=f _(c)[b−1]+ERBToHz(HzToERB(f _(c)[b−1])+Δ)b=2 . . . B  (3b)f _(c)[B]<f _(max),  (3c)where Δ is the desired ERB spacing of the Analysis Filterbank 100, andwhere f_(min) and f_(max) are the desired minimum and maximum centerfrequencies, respectively. One may choose Δ=1, and taking into accountthe frequency range over which the human ear is sensitive, one may setf_(min)=50 Hz and f_(max)=20,000 Hz. With such parameters, for example,application of Eqns. 3a-c yields B=40 auditory filters.

The magnitude frequency response of each auditory filter may becharacterized by a rounded exponential function, as suggested by Mooreand Glasberg. Specifically, the magnitude response of a filter withcenter frequency f_(c)[b] may be computed as:

$\begin{matrix}{{H_{b}(f)} = {\left( {1 + {pg}} \right)e^{- {pg}}}} & \left( {4a} \right) \\{where} & \; \\{{g = {\frac{f - {f_{c}\lbrack b\rbrack}}{f_{c}\lbrack b\rbrack}}},} & \left( {4b} \right) \\{p = \frac{4{f_{c}\lbrack b\rbrack}}{{ERB}\left( {f_{c}\lbrack b\rbrack} \right)}} & \left( {4c} \right)\end{matrix}$The magnitude responses of such B auditory filters, which approximatecritical banding on the ERB scale, are shown in FIG. 10.

The filtering operations of Analysis Filterbank 100 may be adequatelyapproximated using a finite length Discrete Fourier Transform, commonlyreferred to as the Short-Time Discrete Fourier Transform (STDFT),because an implementation running the filters at the sampling rate ofthe audio signal, referred to as a full-rate implementation, is believedto provide more temporal resolution than is necessary for accurateloudness measurements. By using the STDFT instead of a full-rateimplementation, an improvement in efficiency and reduction incomputational complexity may be achieved.

The STDFT of input audio signal x[n] is defined as:

$\begin{matrix}{{{X\left\lbrack {k,t} \right\rbrack} = {\sum\limits_{n = 0}^{N - 1}{{w\lbrack n\rbrack}{x\left\lbrack {n + {tT}} \right\rbrack}e^{{- j}\frac{2\;{\pi k}}{N}}}}},} & \left( {5a} \right)\end{matrix}$where k is the frequency index, t is the time block index, N is the DFTsize, T is the hop size, and w[n] is a length N window normalized sothat

$\begin{matrix}{{\sum\limits_{n = 0}^{N - 1}{w^{2}\lbrack n\rbrack}} = 1} & \left( {5b} \right)\end{matrix}$

Note that the variable t in Eqn. 5a is a discrete index representing thetime block of the STDFT as opposed to a measure of time in seconds. Eachincrement in t represents a hop of T samples along the signal x[n].Subsequent references to the index t assume this definition. Whiledifferent parameter settings and window shapes may be used dependingupon the details of implementation, for f_(s)=44100 Hz, choosing N=2048,T=1024, and having w[n] be a Hanning window provides an adequate balanceof time and frequency resolution. The STDFT described above may be moreefficient using the Fast Fourier Transform (FFT).

Instead of the STDFT, the Modified Discrete Cosine Transform (MDCT) maybe utilized to implement the analysis filterbank. The MDCT is atransform commonly used in perceptual audio coders, such as Dolby AC-3.If the disclosed system is implemented with such perceptually codedaudio, the disclosed loudness measurement and modification may be moreefficiently implemented by processing the existing MDCT coefficients ofthe coded audio, thereby eliminating the need to perform the analysisfilterbank transform. The MDCT of the input audio signal x[n] is givenby:

$\begin{matrix}{{{X\left\lbrack {k,t} \right\rbrack} = {\sum\limits_{n = 0}^{N - 1}{{w\lbrack n\rbrack}{x\left\lbrack {n + {tT}} \right\rbrack}{\cos\left( {\left( {2{\pi/N}} \right)\left( {k + {1/2}} \right)\left( {n + n_{0}} \right)} \right)}}}},{{{where}\mspace{14mu} n_{0}} = \frac{\left( {N/2} \right) + 1}{2}}} & (6)\end{matrix}$

Generally, the hopsize T is chosen to be exactly one-half the transformlength N so that perfect reconstruction of the signal x[n] is possible.

Transmission Filter 101

The outputs of Analysis Filterbank 100 are applied to a transmissionfilter or transmission filter function (“Transmission Filter”) 101 whichfilters each band of the filterbank in accordance with the transmissionof audio through the outer and middle ear. FIG. 8 depicts one suitablemagnitude frequency response of the transmission filter, P(f), acrossthe audible frequency range. The response is unity below 1 kHz, and,above 1 kHz, follows the inverse of the threshold of hearing asspecified in the IS0226 standard, with the threshold normalized to equalunity at 1 kHz.

Excitation 102

In order to compute the loudness of the input audio signal, a measure ofthe audio signals' short-time energy in each filter of the AnalysisFilterbank 100 after application of the Transmission Filter 101 isneeded. This time and frequency varying measure is referred to as theexcitation. The short-time energy output of each filter in AnalysisFilterbank 100 may be approximated in Excitation Function 102 throughmultiplication of filter responses in the frequency domain with thepower spectrum of the input signal:

$\begin{matrix}{{{E\left\lbrack {b,t} \right\rbrack} = {\frac{1}{N}{\sum\limits_{k = 0}^{N - 1}{{{H_{b}\lbrack k\rbrack}}^{2}{{P\lbrack k\rbrack}}^{2}{{X\left\lbrack {k,t} \right\rbrack}}^{2}}}}},} & (7)\end{matrix}$where b is the band number, t is the block number, and H_(b)[k] and P[k]are the frequency responses of the auditory filter and transmissionfilter, respectively, sampled at a frequency corresponding to STDFT orMDCT bin index k. It should be noted that forms for the magnituderesponse of the auditory filters other than that specified in Eqns. 4a-cmay be used in Eqn. 7 to achieve similar results. For example, saidInternational Application No. PCT/US2004/016964, published as WO2004/111964 A2, describes two alternatives: an auditory filtercharacterized by a 12^(th) order IIR transfer function, and a low-cost“brick-wall” band pass approximation.

In summary, the output of Excitation Function 102 is a frequency domainrepresentation of energy E in respective ERB bands b per time period t.

Time Averaging (“Smoothing”) 103

For certain applications of the disclosed invention, as described below,it may be desirable to smooth the excitation E[b,t] prior to itstransformation to specific loudness. For example, smoothing may beperformed recursively in Smoothing function 103 according to theequation:Ē[b,t]=λ_(b) Ē[b,t]+(1−λ_(b))E[b,t],  (8)where the time constants λ_(b) at each band b are selected in accordancewith the desired application. In most cases the time constants may beadvantageously chosen to be proportionate to the integration time ofhuman loudness perception within band b. Watson and Gengel performedexperiments demonstrating that this integration time is within the rangeof 150-175 ms at low frequencies (125-200 Hz) and 40-60 ms at highfrequencies (Charles S. Watson and Roy W. Gengel, “Signal Duration andSignal Frequency in Relation to Auditory Sensitivity” Journal of theAcoustical Society of America, Vol. 46, No. 4 (Part 2), 1969, pp.989-997).

Specific Loudness 104

In the specific loudness converter or conversion function (“SpecificLoudness”) 104, each frequency band of the excitation is converted intoa component value of the specific loudness, which is measured in soneper ERB.

Initially, in computing specific loudness, the excitation level in eachband of Ē[b,t] may be transformed to an equivalent excitation level at 1kHz as specified by the equal loudness contours of ISO 226 (FIG. 11)normalized by the transmission filter P(z) (FIG. 12):Ē _(1kHz):[b,t]=T _(1kHz)(Ē[b,t],f _(c)[b]),  (9)where T_(1kHz) (E,f) is a function that generates the level at 1 kHz,which is equally loud to level E at frequency f. In practice, T_(1kHz)(E,f) is implemented as an interpolation of a look-up table of the equalloudness contours, normalized by the transmission filter. Transformationto equivalent levels at 1 kHz simplifies the following specific loudnesscalculation.

Next, the specific loudness in each band may be computed as:N[b,t]=α[b,t]N _(NB)[b,t]+(1−α[b,t])N _(WB)[b,t],  (10)where N_(NB)[b,t] and N_(WB)[b,t] are specific loudness values based ona narrowband and wideband signal model, respectively. The value α[b,t]is an interpolation factor lying between 0 and 1 that is computed fromthe audio signal. Said International Application No. PCT/US2004/016964,published as WO 2004/111964 A2, describes a technique for calculatingα[b,t] from the spectral flatness of the excitation. It also describes“narrowband” and “wideband” signal models in greater detail.

The narrowband and wideband specific loudness values N_(NB)[b,t] andN_(WB)[b, t] may be estimated from the transformed excitation using theexponential functions:

$\begin{matrix}{{N_{NB}\left\lbrack {b,t} \right\rbrack} = \left\{ \begin{matrix}{{G_{NB}\left( {\left( \frac{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack}{{TQ}_{1\mspace{14mu}{kHz}}} \right)^{\beta_{NB}} - 1} \right)},} & {{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack} > {TQ}_{11\mspace{14mu}{kHz}}} \\{0,} & {otherwise}\end{matrix} \right.} & \left( {11a} \right) \\{{N_{WB}\left\lbrack {m,t} \right\rbrack} = \left\{ {\begin{matrix}{G_{WB}\left( {\left( {\left( \frac{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack}{{TQ}_{1\mspace{14mu}{kHz}}} \right)^{\beta_{WB}} - 1} \right),} \right.} & {{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack} > {TQ}_{1\mspace{14mu}{kHz}}} \\{0,} & {otherwise}\end{matrix},} \right.} & \left( {11b} \right)\end{matrix}$where TQ_(1kHz) is the excitation level at threshold in quiet for a 1kHz tone. From the equal loudness contours (FIGS. 11 and 12) TQ_(1kHz)equals 4.2 dB. One notes that both of these specific loudness functionsare equal to zero when the excitation is equal to the threshold inquiet. For excitations greater than the threshold in quiet, bothfunctions grow monotonically with a power law in accordance withStevens' law of intensity sensation. The exponent for the narrowbandfunction is chosen to be larger than that of the wideband function,making the narrowband function increase more rapidly than the widebandfunction. The specific selection of exponents β and gains G for thenarrowband and wideband cases and are chosen to match experimental dataon the growth of loudness for tones and noise.

Moore and Glasberg suggest that the specific loudness should be equal tosome small value instead of zero when the excitation is at the thresholdof hearing. Specific loudness should then decrease monotonically to zeroas the excitation decreases to zero. The justification is that thethreshold of hearing is a probabilistic threshold (the point at which atone is detected 50% of the time), and that a number of tones, each atthreshold, presented together may sum to a sound that is more audiblethan any of the individual tones. In the disclosed application,augmenting the specific loudness functions with this property has theadded benefit of making the gain solver, discussed below, behave moreappropriately when the excitation is near threshold. If the specificloudness is defined to be zero when the excitation is at or belowthreshold, then a unique solution for the gain solver does not exist forexcitations at or below threshold. If, on the other hand, specificloudness is defined to be monotonically increasing for all values ofexcitation greater than or equal to zero, as suggested by Moore andGlasberg, then a unique solution does exist. Loudness scaling greaterthan unity will always result in a gain greater than unity and viceversa. The specific loudness functions in Eqns. 11a and 11b may bealtered to have the desired property according to:

$\begin{matrix}{{N_{NB}\left\lbrack {b,t} \right\rbrack} = \left\{ \begin{matrix}{{G_{NB}\left( {\left( \frac{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack}{{TQ}_{1\mspace{14mu}{kHz}}} \right)^{\beta_{NB}} - 1} \right)},} & {{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack} > {\lambda\;{TQ}_{11\mspace{14mu}{kHz}}}} \\{\exp\left\{ {{K_{NB}\left( {- {\log\left( {\left( \frac{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack}{{TQ}_{1\mspace{14mu}{kHz}}} \right) + C_{NB}} \right)}^{\eta_{NB}}} \right\}},} \right.} & {otherwise}\end{matrix} \right.} & \left( {11a} \right) \\{{N_{WB}\left\lbrack {m,t} \right\rbrack} = \left\{ {\begin{matrix}{G_{WB}\left( {\left( {\left( \frac{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack}{{TQ}_{1\mspace{14mu}{kHz}}} \right)^{\beta_{WB}} - 1} \right),} \right.} & {{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack} > {\lambda\;{TQ}_{1\mspace{14mu}{kHz}}}} \\{\exp\left\{ {{K_{WB}\left( {- {\log\left( {\left( \frac{{\overset{\_}{E}}_{1\mspace{14mu}{kHz}}\left\lbrack {b,t} \right\rbrack}{{TQ}_{1\mspace{14mu}{kHz}}} \right) + C_{WB}} \right)}^{\eta_{WB}}} \right\}},} \right.} & {otherwise}\end{matrix},} \right.} & \left( {11b} \right)\end{matrix}$where the constant λ is greater than one, the exponent η is less thanone, and the constants K and C are chosen so that the specific loudnessfunction and its first derivative are continuous at the pointĒ_(1kHz)[b,t]=λTQ_(1kHz).

From the specific loudness, the overall or “total” loudness L[t] isgiven by the sum of the specific loudness across all bands b:

$\begin{matrix}{{L\lbrack t\rbrack} = {\sum\limits_{b}{N\left\lbrack {b,t} \right\rbrack}}} & (12)\end{matrix}$

Specific Loudness Modification 105

In the specific loudness modification function (“Specific LoudnessModification”) 105, the target specific loudness, referred to as{circumflex over (N)}[b,t], may be calculated from the specific loudnessof SL 104 (FIG. 7) in various ways depending on the desired applicationof the overall device or process. As is described in greater detailbelow, a target specific loudness may be calculated using a scale factorα, for example, in the case of a volume control. See Eqn. 16 below andits associated description. In the case of automatic gain control (AGC)and dynamic range control (DRC), a target specific loudness may becalculated using a ratio of desired output loudness to input loudness.See Eqns. 17 and 18 below and their associated descriptions. In the caseof dynamic equalization, a target specific loudness may be calculatedusing a relationship set forth in Eqn. 23 and its associateddescription.

Gain Solver 106

In this example, for each band b and every time interval t, the GainSolver 106 takes as its inputs the smoothed excitation Ē[b,t] and thetarget specific loudness {circumflex over (N)}[b, t] and generates gainsG[b,t] used subsequently for modifying the audio. Letting the functionΨ{⋅} represent the non-linear transformation from excitation to specificloudness such thatN[b,t]=Ψ{Ē[b,t]},  (13)the Gain Solver finds G[b,t] such that{circumflex over (N)}[b,t]=Ψ{G ²[b,t]Ē[b,t]}.  (14a)

The Gain Solvers 106 determine frequency- and time-varying gains, which,when applied to the original excitation, result in a specific loudnessthat, ideally, is equal to the desired target specific loudness. Inpractice, the Gain Solvers determine frequency- and time-varying gains,which when applied to the frequency-domain version of the audio signalresults in modifying the audio signal in order to reduce the differencebetween its specific loudness and the target specific loudness. Ideally,the modification is such that the modified audio signal has a specificloudness that is a close approximation of the target specific loudness.The solution to Eqn. 14a may be implemented in a variety of ways. Forexample, if a closed form mathematical expression for the inverse of thespecific loudness, represented by Ψ⁻¹ {⋅}, exists, then the gains may becomputed directly by re-arranging equation 14a:

$\begin{matrix}{{G\left\lbrack {b,t} \right\rbrack} = \sqrt{\frac{\Psi^{- 1}\left( {\hat{N}\left\lbrack {b,t} \right\rbrack} \right)}{\overset{\_}{E}\left\lbrack {b,t} \right\rbrack}}} & \left( {14b} \right)\end{matrix}$

Alternatively, if a closed form solution for Ψ⁻¹{⋅} does not exist, aniterative approach may be employed in which for each iteration equation14a is evaluated using a current estimate of the gains. The resultingspecific loudness is compared with the desired target and the gains areupdated based on the error. If the gains are updated properly, they willconverge to the desired solution. Another method involves pre-computingthe function Ψ{⋅} for a range of excitation values in each band tocreate a look-up table. From this look-up table, one obtains anapproximation of the inverse function Ψ⁻¹{⋅} and the gains may then becomputed from equation 14b. As mentioned earlier, the target specificloudness may be represented by a scaling of the specific loudness:{circumflex over (N)}[b,t]=Ξ[b,t]N[b,t]  (14c)

Substituting equation 13 into 14c and then 14c into 14b yields analternative expression for the gains:

$\begin{matrix}{{G\left\lbrack {b,t} \right\rbrack} = \sqrt{\frac{\Psi^{- 1}\left( {{\Xi\left\lbrack {b,t} \right\rbrack}{\Psi\left( {\overset{\_}{E}\left\lbrack {b,t} \right\rbrack} \right)}} \right)}{\overset{\_}{E}\left\lbrack {b,t} \right\rbrack}}} & \left( {14d} \right)\end{matrix}$

We see that the gains may be expressed purely as a function of theexcitation Ē[b,t] and the specific loudness scaling Ξ[b,t]. Therefore,the gains may be computed through evaluation of 14 d or an equivalentlookup table without ever explicitly computing the specific loudness ortarget specific loudness as intermediate values. However, these valuesare implicitly computed through use of equation 14d. Other equivalentmethods for computing the modification parameters through eitherexplicit or implicit computation of the specific loudness and targetspecific loudness may be devised, and this invention is intended tocover all such methods.

Synthesis Filterbank 110

As described above, Analysis Filterbank 100 may be implementedefficiently through use of the Short-time Discrete Fourier Transform(STDFT) or the Modified Discrete Cosine Transform, and the STDFT or MDCTmay be used similarly to implement Synthesis Filterbank 110.Specifically, letting X[k, t] represent the STDFT or MDCT of the inputaudio, as defined earlier, the STDFT or MDCT of the processed (modified)audio in Synthesis Filterbank 110 may be calculated as

$\begin{matrix}{{{\hat{X}\left\lbrack {k,t} \right\rbrack} = {\sum\limits_{b}{{G\left\lbrack {b,t} \right\rbrack}{S_{b}\lbrack k\rbrack}{X\left\lbrack {k,{t - d}} \right\rbrack}}}},} & (15)\end{matrix}$where S_(b)[k] is the response of the synthesis filter associated withband b, and d is the delay associated with delay block 109 in FIG. 7.The shape of the synthesis filters S_(b)[k] may be chosen the same asthe filters utilized in the analysis filterbank, H_(b)[k], or they maybe modified to provide perfect reconstruction in the absence of any gainmodification (i.e., when G[b,t]=1). The final processed audio may thenbe generated through inverse Fourier or modified cosine transform of{circumflex over (X)}[k, t] and overlap-add synthesis, as is familiar toone skilled in the art.

Target Specific Loudness

The behavior of arrangements embodying aspects of the invention such asthe examples of FIGS. 1-7 is dictated mainly by the manner in which thetarget specific loudness {circumflex over (N)}[b,t] is calculated.Although the invention is not limited by any particular function orinverse function for calculating target specific loudness, several suchfunctions and suitable applications for them will now be described.

Time-Invariant and Frequency-Invariant Function Suitable for VolumeControl

A standard volume control adjusts the loudness of an audio signal byapplying a wideband gain to the audio. Generally, the gain is coupled toa knob or slider that is adjusted by a user until the loudness of theaudio is at the desired level. An aspect of the present invention allowsfor a more psychoacoustically consistent way of implementing such acontrol. According to this aspect of the invention, rather than having awideband gain coupled to the volume control that results in a change ofgain by the same amount across all frequency bands, which may cause achange in the perceived spectrum, a specific loudness scaling factor isassociated with the volume control adjustment instead so that the gainin each of multiple frequency bands is changed by an amount that takesinto account the human hearing model so that, ideally, there is nochange in the perceived spectrum. In the context of this aspect of theinvention and an exemplary application thereof, “constant” or“time-invariant” is intended to allow for changes in the setting of avolume control scale factor from time to time, for example, by a user.Such “time-invariance” is sometimes referred to as “quasitime-invariant,” “quasi-stationary,” “piecewise time-invariant,”“piecewise stationary,” “step-wise time-invariant,” and “step-wisestationary.” Given such a scale factor, α, the target specific loudnessmay be calculated as the measured specific loudness multiplied by a:{umlaut over (N)}[b,t]=αN[b,t].  (16)

Because total loudness L[t] is the sum of specific loudness N[b,t]across all bands b, the above modification also scales the totalloudness by a factor of α, but it does so in a way that preserves thesame perceived spectrum at a particular time for changes in the volumecontrol adjustment. In other words, at any particular time, a change inthe volume control adjustment results in a change in perceived loudnessbut no change in the perceived spectrum of the modified audio versus theperceived spectrum of the unmodified audio. FIG. 13a depicts theresulting multiband gains G[b,t] across the bands “b” at a particulartime “t” when α=0.25 for an audio signal consisting of female speech.For comparison, the wideband gain required to scale the original totalloudness by 0.25 (the horizontal line), as in a standard volume control,is also plotted. The multiband gain G[b,t] increases at low and highfrequency bands in comparison to the middle frequency bands. This isconsistent with equal-loudness contours indicating that the human ear isless sensitive at low and high frequencies.

FIG. 13b depicts the specific loudness for the original audio signal,the wideband gain-modified signal as modified in accordance with a priorart volume control, and the multiband gain-modified signal as modifiedin accordance with this aspect of the invention. The specific loudnessof the multiband gain modified signal is that of the original scaled by0.25. The specific loudness of the wide-band gain modified signal haschanged its spectral shape with respect to that of the originalunmodified signal. In this case, the specific loudness has, in arelative sense, lost loudness at both the low and the high frequencies.This is perceived as a dulling of the audio as its volume is turneddown, a problem that does not occur with the multiband modified signalwhose loudness is controlled by gains derived in the perceptual loudnessdomain.

Along with the distortion of the perceived spectral balance associatedwith a traditional volume control there exists a second problem. Aproperty of loudness perception, which is reflected in the loudnessmodel reflected in Equations 11a-11d, is that loudness of a signal atany frequency decreases more rapidly as signal level approaches thethreshold of hearing. As a result, the electrical attenuation requiredto impart the same loudness attenuation to a softer signal is less thanthat required for a louder signal. A traditional volume control impartsa constant attenuation regardless of signal level, and therefore softsignals become “too soft” with respect to louder signals as the volumeis turned down. In many cases this results in the loss of detail in theaudio. Consider the recording of a castanet in a reverberant room. Insuch a recording the main “hit” of the castanet is quite loud incomparison to the reverberant echoes, but it is the reverberant echoesthat convey the size of the room. As the volume is turned down with atraditional volume control, the reverberant echoes become softer withrespect to the main hit and eventually disappear below the threshold ofhearing, leaving a “dry” sounding castanet. The loudness based volumecontrol prevents the disappearance of the softer portions of therecordings by boosting the softer reverberant portion of the recordingrelative to the louder main hit so that the relative loudness betweenthese sections remains constant. In order to achieve this effect, themultiband gains G[b,t] must vary over time at a rate that iscommensurate with the human temporal resolution of loudness perception.Because the multiband gains G[b,t] are computed as a function of thesmoothed excitation Ē[b,t], selection of the time constants λ_(b) inEqn. 8 dictates how quickly the gains may vary across time in each bandb. As mentioned earlier, these time constants may be selected to beproportionate the integration time of human loudness perception withinband b and thus yield the appropriate variation of G[b,t] over time. Itshould be noted that if the time constants are chosen inappropriately(either too fast or too slow), then perceptually objectionable artifactsmay be introduced in the processed audio.

Time-Invariant and Frequency-Variant Function Suitable for FixedEqualization

In some applications, one may wish to apply a fixed perceptualequalization to the audio, in which case the target specific loudnessmay be computed by applying a time-invariant but frequency-variant scalefactor Θ[b] as in the relationship{circumflex over (N)}[b,t]=Θ[b]N[b,t],wherein {circumflex over (N)}[b,t] is the target specific loudness,N[b,t] is the specific loudness of the audio signal, b is a measure offrequency, and t is a measure of time. In this case, the scaling mayvary from band to band. Such an application may be useful foremphasizing, for example, the portion of the spectrum dominated byspeech frequencies in order to boost intelligibility.

Frequency-Invariant and Time-Variant Function Suitable for AutomaticGain and Dynamic Range Control

The techniques of automatic gain and dynamic range control (AGC and DRC)are well known in the audio processing field. In an abstract sense, bothtechniques measure the level of an audio signal in some manner and thengain-modify the signal by an amount that is a function of the measuredlevel. For the case of AGC, the signal is gain-modified so that itsmeasured level is closer to a user selected reference level. With DRC,the signal is gain-modified so that the range of the signal's measuredlevel is transformed into some desired range. For example, one may wishto make the quiet portions of the audio louder and the loud portionsquieter. Such a system is described by Robinson and Gundry (CharlesRobinson and Kenneth Gundry, “Dynamic Range Control via Metadata,”107^(th) Convention of the AES, Preprint 5028, Sep. 24-27, 1999, NewYork). Traditional implementations of AGC and DRC generally utilize asimple measurement of audio signal level, such as smoothed peak or rootmean square (rms) amplitude, to drive the gain modification. Such simplemeasurements correlate to some degree to the perceived loudness of theaudio, but aspects of the present invention allow for more perceptuallyrelevant AGC and DRC by driving the gain modifications with a measure ofloudness based on a psychoacoustic model. Also, many traditional AGC andDRC systems apply the gain modification with a wideband gain, therebyincurring the aforementioned timbral (spectral) distortions in theprocessed audio. Aspects of the present invention, on the other hand,utilize a multiband gain to shape the specific loudness in a manner thatreduces or minimizes such distortions.

Both the AGC and DRC applications employing aspects of the presentinvention are characterized by a function that transforms or maps aninput wideband loudness L_(i)[t] into a desired output wideband loudnessL_(o)[t], where the loudness is measured in perceptual loudness units,such as sone. The input wideband loudness L_(i)[t] is a function of theinput audio signal's specific loudness N[b,t]. Although it may be thesame as the input audio signal's total loudness, it may be atemporally-smoothed version of the audio signal's total loudness.

FIGS. 14a and 14b depict examples of mapping functions typical for anAGC and a DRC, respectively. Given such a mapping in which L_(o)[t] is afunction of L_(i)[t], the target specific loudness may be calculated as

$\begin{matrix}{{\hat{N}\left\lbrack {b,t} \right\rbrack} = {\frac{L_{o}\lbrack t\rbrack}{L_{i}\lbrack t\rbrack}{{N\left\lbrack {b,t} \right\rbrack}.}}} & (17)\end{matrix}$

The audio signal's original specific loudness N[b,t] is simply scaled bythe ratio of the desired output wideband loudness to the input widebandloudness to yield an output specific loudness {circumflex over(N)}[b,t]. For an AGC system, the input wideband loudness L_(i)[t]should generally be a measure of the long-term total loudness of theaudio. This can be achieved by smoothing the total loudness L[t] acrosstime to generate L_(i)[t].

In comparison to an AGC, a DRC system reacts to shorter term changes ina signal's loudness, and therefore L_(i)[t] can simply be made equal toL[t]. As a result, the scaling of specific loudness, given byL_(o)[t]/L_(i)[t], may fluctuate rapidly leading to unwanted artifactsin the processed audio. One typical artifact is the audible modulationof a portion of the frequency spectrum by some other relativelyunrelated portion of the spectrum. For example, a classical musicselection might contain high frequencies dominated by a sustained stringnote, while the low frequencies contain a loud booming timpani. Wheneverthe timpani hits, the overall loudness L_(i)[t] increases, and the DRCsystem applies attenuation to the entire specific loudness. The stringsare then heard to “pump” down and up in loudness with the timpani. Suchcross pumping in the spectrum is a problem with traditional wideband DRCsystems as well, and a typical solution involves applying DRCindependently to different frequency bands. The system disclosed here isinherently multiband due to the filterbank and the calculation ofspecific loudness that employs a perceptual loudness model, andtherefore modifying a DRC system to operate in a multiband fashion inaccordance with aspects of the present invention is relativelystraightforward and is next described.

Frequency-Variant and Time-Variant Function Suitable for Dynamic RangeControl

The DRC system may be expanded to operate in a multiband orfrequency-variant fashion by allowing the input and output loudness tovary independently with band b. These multiband loudness values arereferenced as L_(i)[b,t] and L_(o)[b,t], and the target specificloudness may then be given by

$\begin{matrix}{{{\hat{N}\left\lbrack {b,t} \right\rbrack} = {\frac{L_{o}\left\lbrack {b,t} \right\rbrack}{L_{i}\left\lbrack {b,t} \right\rbrack}{N\left\lbrack {b,t} \right\rbrack}}},} & (18)\end{matrix}$where L_(o)[b,t] has been calculated from or mapped from L_(i)[b,t], asillustrated in FIG. 14b , but independently for each band b. The inputmultiband loudness L_(i)[b,t] is a function of the input audio signal'sspecific loudness N[b,t]. Although it may be the same as the input audiosignal's specific loudness, it may be a temporally-smoothed and/orfrequency-smoothed version of the audio signal's specific loudness.

The most straightforward way of calculating L_(i)[b,t] is to set itequal to the specific loudness N[b,t]. In this case, DRC is performedindependently on every band in the auditory filterbank of the perceptualloudness model rather than in accordance with the same input versusoutput loudness ratio for all bands as just described above under theheading “Frequency-Invariant and Time-Variant Function Suitable forAutomatic Gain and Dynamic Range Control.” In a practical embodimentemploying 40 bands, the spacing of these bands along the frequency axisis relatively fine in order to provide an accurate measure of loudness.However, applying a DRC scale factor independently to each band maycause the processed audio to sound “torn apart”. To avoid this problem,one may choose to calculate L_(i)[b,t] by smoothing specific loudnessN[b,t] across bands so that the amount of DRC applied from one band tothe next does not vary as drastically. This may be achieved by defininga band-smoothing filter Q(b) and then smoothing the specific loudnessacross all bands c according to the standard convolution sum:

$\begin{matrix}{{L_{i}\left\lbrack {b,t} \right\rbrack} = {\sum\limits_{c}{{Q\left( {b - c} \right)}{{N\left\lbrack {c,t} \right\rbrack}.}}}} & (19)\end{matrix}$wherein N[c,t] is the specific loudness of the audio signal and Q(b−c)is the band-shifted response of the smoothing filter. FIG. 15 depictsone example of such a band-smoothing filter.

If the DRC function that calculates L_(i)[b,t] as a function ofL_(o)[b,t] is fixed for every band b, then the type of change incurredto each band of the specific loudness N[b,t] will vary depending on thespectrum of the audio being processed, even if the overall loudness ofthe signal remains the same. For example, an audio signal with loud bassand quiet treble may have the bass cut and the treble boosted. A signalwith quiet bass and loud treble may have the opposite occur. The neteffect is a change in the timbre or perceived spectrum of the audio, andthis may be desirable in certain applications.

However, one may wish to perform multiband DRC without modifying theaverage perceived spectrum of the audio. One might want the averagemodification in each band to be roughly the same while still allowingthe short-term variations of the modifications to operate independentlybetween and among bands. The desired effect may be achieved by forcingthe average behavior of the DRC in each band to be the same as that ofsome reference behavior. One may choose this reference behavior as thedesired DRC for the wideband input loudness L_(i)[t]. Let the functionL_(o)[t]=DRC{L_(i)[t]}represent the desired DRC mapping for the widebandloudness. Then let L _(i)[t] represent a time-averaged version of thewideband input loudness, and let L _(i)[b,t] represent a time-averagedversion of the multiband input loudness L_(i)[b,t]. The multiband outputloudness may then be calculated as

$\begin{matrix}{{L_{o}\left\lbrack {b,t} \right\rbrack} = {\frac{{\overset{\_}{L}}_{i}\left\lbrack {b,t} \right\rbrack}{{\overset{\_}{L}}_{i}\lbrack t\rbrack}{DRC}{\left\{ {\frac{{\overset{\_}{L}}_{i}\lbrack t\rbrack}{{\overset{\_}{L}}_{i}\left\lbrack {b,t} \right\rbrack}{{\overset{\_}{L}}_{i}\left\lbrack {b,t} \right\rbrack}} \right\}.}}} & (20)\end{matrix}$

Note that the multiband input loudness is first scaled to be in the sameaverage range as the wideband input loudness. The DRC function designedfor the wideband loudness is then applied. Lastly, the result is scaledback down to the average range of the multiband loudness. With thisformulation of multiband DRC, the benefits of reduced spectral pumpingare retained, while at the same time preserving the average perceivedspectrum of the audio.

Frequency-Variant and Time-Variant Function Suitable for DynamicEqualization

Another application of aspects of the present invention is theintentional transformation of the audio's time-varying perceivedspectrum to a target time-invariant perceived spectrum while stillpreserving the original dynamic range of the audio. One may refer tothis processing as Dynamic Equalization (DEQ). With traditional staticequalization, a simple fixed filtering is applied to the audio in orderto change its spectrum. For example, one might apply a fixed bass ortreble boost. Such processing does not take into account the currentspectrum of the audio and may therefore be inappropriate for somesignals, i.e., signals that already contain a relatively large amount ofbass or treble. With DEQ, the spectrum of the signal is measured and thesignal is then dynamically modified in order to transform the measuredspectrum into an essentially static desired shape. For aspects of thepresent invention, such a desired shape is specified across bands in thefilterbank and referred to as EQ[b]. In a practical embodiment, themeasured spectrum should represent the average spectral shape of theaudio that may be generated by smoothing the specific loudness N[b,t]across time. One may refer to the smoothed specific loudness as N[b,t].As with the multiband DRC, one may not want the DEQ modification to varydrastically from one band to the next, and therefore a band-smoothingfunction may be applied to generate a band-smoothed spectrum L[b,t]:

$\begin{matrix}{{\overset{\_}{L}\left\lbrack {b,t} \right\rbrack} = {\sum\limits_{c}{{Q\left( {b - c} \right)}{{\overset{\_}{N}\left\lbrack {c,t} \right\rbrack}.}}}} & (21)\end{matrix}$

In order to preserve the original dynamic range of the audio, thedesired spectrum EQ[b] should be normalized to have the same overallloudness as the measured spectral shape given by L[b,t]. One may referto this normalized spectral shape as L _(EQ)[b,t]:

$\begin{matrix}{{{\overset{\_}{L}}_{EQ}\left\lbrack {b,t} \right\rbrack} = {\left( \frac{\sum\limits_{c}{\overset{\_}{L}\left\lbrack {c,t} \right\rbrack}}{\sum\limits_{c}{{EQ}\lbrack c\rbrack}} \right){{{EQ}\lbrack b\rbrack}.}}} & (22)\end{matrix}$

Finally, the target specific loudness is calculated as

$\begin{matrix}{{{\hat{N}\left\lbrack {b,t} \right\rbrack} = {\left( \frac{\overset{\_}{L}\left\lbrack {b,t} \right\rbrack}{{\overset{\_}{L}}_{EQ}\left\lbrack {b,t} \right\rbrack} \right)^{1 - \beta}\frac{{\overset{\_}{L}}_{EQ}\left\lbrack {b,t} \right\rbrack}{\overset{\_}{L}\left\lbrack {b,t} \right\rbrack}{N\left\lbrack {b,t} \right\rbrack}}},} & (23)\end{matrix}$where β is a user-specified parameter ranging from zero to one,indicating the degree of DEQ that is to be applied. Looking at Eqn. 23,one notes that when β=0, the original specific loudness is unmodified,and when β=1, the specific loudness is scaled by the ratio of thedesired spectral shape to the measured spectral shape.

One convenient way of generating the desired spectral shape EQ[b] is fora user to set it equal to L[b,t] as measured for some piece of audiowhose spectral balance the user finds pleasing. In a practicalembodiment, for example as shown in FIG. 16, the user may be provided abutton or other suitable actuator 507 that, when actuated, causes acapture of the current measure of the audio's spectral shape L[b,t], andthen stores this measure as a preset (in Target Specific Loudness PresetCapture and Store 506) that may later be loaded into EQ[b] when DEQ isenabled (as by preset select 508). FIG. 16 is a simplified version ofFIG. 7 in which only a single line is shown to represent multiple bandsfrom Analysis Filterbank 100 to Synthesis Filterbank 110. The FIG. 17example also provides a Dynamic EQ Specific Loudness (SL) Modification505 that provides a modification to the specific loudness measured byfunction or device 104 in accordance with dynamic equalization, asexplained above.

Combined Processing

One may wish to combine all the previously described processing,including Volume Control (VC), AGC, DRC, and DEQ, into a single system.Because each of these processes may be represented as a scaling of thespecific loudness, all of them are easily combined as follows:{circumflex over(N)}[b,t]=(Ξ_(VC)[b,t]Ξ_(AGC)[b,t]Ξ_(DRC)[b,t]Ξ_(DEQ)[b,t])N[b,t],  (24)where Ξ_(*)[b,t] represents the scale factors associated with process“*”. A single set of gains G[b,t] may then be calculated for the targetspecific loudness that represents the combined processing.

In some cases, the scale factors of one or a combination of the loudnessmodification processes may fluctuate too rapidly over time and produceartifacts in the resulting processed audio. It may therefore bedesirable to smooth some subset of these scaling factors. In general,the scale factors from VC and DEQ varying smoothly over time, butsmoothing the combination of the AGC and DRC scale factors may berequired. Let the combination of these scale factors be represented byΞ_(C)[b,t]=Ξ_(AGC)[b,t]Ξ_(DRC)[b,t]  (25)

The basic notion behind the smoothing is that the combined scale factorsshould react quickly when the specific loudness is increasing, and thatthe scale factors should be more heavily smoothed when the specificloudness is decreasing. This notion corresponds to the well-knownpractice of utilizing a fast attack and a slow release in the design ofaudio compressors. The appropriate time constants for smoothing thescale factors may be calculated by smoothing across time a band-smoothedversion of the specific loudness. First a band-smoothed version of thespecific loudness is computed:

$\begin{matrix}{{L\left\lbrack {b,t} \right\rbrack} = {\sum\limits_{c}{{Q\left( {b - c} \right)}{{N\left\lbrack {c,t} \right\rbrack}.}}}} & (26)\end{matrix}$wherein N[c,t] is the specific loudness of the audio signal and Q(b−c)is the band-shifted response of the smoothing filter as in Eqn. 19,above.

The time-smoothed version of this band-smoothed specific loudness isthen calculated asL [b,t]=λ[b,t]L[b,t]+(1−λ[b,t]) L [b,t−1]  (27)where the band dependent smoothing coefficient λ[b,t] is given by

$\begin{matrix}{{\lambda\left\lbrack {b,t} \right\rbrack} = \left\{ {\begin{matrix}{\lambda_{fast}}_{,} & {{L\left\lbrack {b,t} \right\rbrack} > {\overset{\_}{L}\left\lbrack {b,t} \right\rbrack}} \\\lambda_{slow} & {{L\left\lbrack {b,t} \right\rbrack} \leq {\overset{\_}{L}\left\lbrack {b,t} \right\rbrack}}\end{matrix}.} \right.} & (28)\end{matrix}$

The smoothed combined scale factors are then calculated asΞ _(C)[b,t]=λ_(M)[b,t]Ξ_(C)[b,t]+(1−λ_(M)[b,t])Ξ _(C)[b,t−1],  (29)where λ_(M) [b,t] is a band-smoothed version of λ[b,t]:

$\begin{matrix}{{\lambda_{M}\left\lbrack {b,t} \right\rbrack} = {\left( \frac{1}{\sum\limits_{c}{Q(c)}} \right){\sum\limits_{c}{{Q\left( {b - c} \right)}{{\lambda\left\lbrack {b,t} \right\rbrack}.}}}}} & (30)\end{matrix}$

Band smoothing of the smoothing coefficient prevents the time-smoothedscale factors from changing drastically across bands. The describedscale factor time- and band-smoothing results in processed audiocontaining fewer objectionable perceptual artifacts.

Noise Compensation

In many audio playback environments there exists background noise thatinterferes with the audio that a listener wishes to hear. For example, alistener in a moving automobile may be playing music over the installedstereo system and noise from the engine and road may significantly alterthe perception of the music. In particular, for parts of the spectrum inwhich the energy of the noise is significant relative to the energy ofthe music, the perceived loudness of the music is reduced. If the levelof the noise is large enough, the music is completely masked. Withrespect to an aspect of the current invention, one would like to choosegains G[b,t] so that the specific loudness of the processed audio in thepresence of the interfering noise is equal to the target specificloudness {circumflex over (N)}[b,t]. To achieve this effect, one mayutilize the concept of partial loudness, as defined by Moore andGlasberg, supra. Assume that one is able to obtain a measurement of thenoise by itself and a measurement of the audio by itself. Let E_(N)[b,t] represent the excitation from the noise and let E_(A)[b,t] representthe excitation from the audio. The combined specific loudness of theaudio and the noise is then given byN _(TOT)[b,t]=Ψ{E _(A)[b,t]+E _(N)[b,t]},  (31)where, again, Ψ{⋅} represents the non-linear transformation fromexcitation to specific loudness. One may assume that a listener'shearing partitions the combined specific loudness between the partialspecific loudness of the audio and the partial specific loudness of thenoise in a way that preserves the combined specific loudness:N _(TOT)[b,t]=N _(A)[b,t]+N _(N)[b,t].  (32)

The partial specific loudness of the audio, N_(A)[b,t], is the value onewishes to control, and therefore one must solve for this value. Thepartial specific loudness of the noise may be approximated as

$\begin{matrix}{{N_{N}\left\lbrack {b,t} \right\rbrack} = {\left( \frac{E_{TN}\left\lbrack {b,t} \right\rbrack}{E_{A}\left\lbrack {b,t} \right\rbrack} \right)^{K}\left( {{\Psi\left\{ {{E_{N}\left\lbrack {b,t} \right\rbrack} + {E_{TN}\left\lbrack {b,t} \right\rbrack}} \right\}} - {\Psi\left\{ {E_{TQ}\lbrack b\rbrack} \right\}}} \right)}} & (33)\end{matrix}$

where E_(TN)[b,t] is the masked threshold in the presence of the noise,E_(TQ)[b] is the threshold of hearing in quiet at band b, and K is anexponent between zero and one.

Combining Eqns. 31-33 one arrives at an expression for the partialspecific loudness of the audio:

$\begin{matrix}{{N_{A}\left\lbrack {b,t} \right\rbrack} = {{\Psi\left\{ {{E_{A}\left\lbrack {b,t} \right\rbrack} + {E_{N}\left\lbrack {b,t} \right\rbrack}} \right\}} - {\left( \frac{E_{TN}\left\lbrack {b,t} \right\rbrack}{E_{A}\left\lbrack {b,t} \right\rbrack} \right)^{K}\left( {{\Psi\left\{ {{E_{N}\left\lbrack {b,t} \right\rbrack} + {E_{TN}\left\lbrack {b,t} \right\rbrack}} \right\}} - {\Psi\left\{ {E_{TQ}\lbrack b\rbrack} \right\}}} \right)}}} & (34)\end{matrix}$

One notes that when the excitation of the audio is equal to the maskedthreshold of the noise (E_(A)[b,t]=E_(TN)[b,t]), the partial specificloudness of the audio is equal to the loudness of a signal at thethreshold in quiet, which is the desired outcome. When the excitation ofthe audio is much greater than that of the noise, the second term inEqn. 34 vanishes, and the specific loudness of the audio isapproximately equal to what it would be if the noise were not present.In other words, as the audio becomes much louder than the noise, thenoise is masked by the audio. The exponent κ is chosen empirically togive a good fit to data on the loudness of a tone in noise as a functionof the signal-to-noise ratio. Moore and Glasberg have found that a valueof κ=0.3 is appropriate. The masked threshold of the noise may beapproximated as a function of the noise excitation itself:E _(TN)[b,t]=K[b]E _(N)[b,t]+E _(TQ)[b]  (35)where K[b] is a constant that increases at lower frequency bands. Thus,the partial specific loudness of the audio given by Eqn. 34 may berepresented abstractly as a function of the excitation of the audio andthe excitation of the noise:N _(A)[b,t]=Φ{E _(A)[b,t],E _(N)[b,t]}.  (36)

A modified gain solver may then be utilized to calculate the gainsG[b,t] such that the partial specific loudness of the processed audio inthe presence of the noise is equal to the target specific loudness:{circumflex over (N)}[b,t]=Φ({G ²[b,t]E _(A)[b,t],E _(N)[b,t]}  (37)

FIG. 17 depicts the system of FIG. 7 with the original Gain Solver 106replaced by the described Noise Compensating Gain Solver 206 (note thatthe multiple vertical lines between blocks representing the multiplebands of the filterbank have been replaced by a single line to simplifythe diagram). In addition, the figure depicts the measurement of thenoise excitation (by Analysis Filterbank 200, Transmission Filter 201,Excitation 202 and Smoothing 203 in a manner corresponding to theoperation of blocks 100, 101, 102 and 103) that feeds into the new gainsolver 206 along with the excitation of the audio (from Smoothing 103)and the target specific loudness (from SL Modification 105).

In its most basic mode of operation, the SL Modification 105 in FIG. 17may simply set the target specific loudness {circumflex over (N)}[b,t]equal to the original specific loudness of the audio N[b,t]. In otherwords, the SL Modification provides a frequency-invariant, scale factorα scaling of the specific loudness of the audio signal, wherein α=1.With an arrangement such as in FIG. 17, the gains are calculated so thatthe perceived loudness spectrum of the processed audio in the presenceof the noise is equal to the loudness spectrum of the audio in theabsence of the noise. Additionally, any one or combination of ones ofthe previously described techniques for computing the target specificloudness as a function of the original, including VC, AGC, DRC, and DEQ,may be utilized in conjunction with the noise compensating loudnessmodification system.

In a practical embodiment, the measurement of the noise may be obtainedfrom a microphone placed in or near the environment into which the audiowill be played. Alternatively, a predetermined set of template noiseexcitations may be utilized that approximate the anticipated noisespectrum under various conditions. For example, the noise in anautomobile cabin may be pre-analyzed at various driving speeds and thenstored as a look-up table of noise excitation versus speed. The noiseexcitation fed into the Gain Solver 206 in FIG. 17 may then beapproximated from this look-up table as the speed of the automobilevaries.

Implementation

The invention may be implemented in hardware or software, or acombination of both (e.g., programmable logic arrays). Unless otherwisespecified, the algorithms included as part of the invention are notinherently related to any particular computer or other apparatus. Inparticular, various general-purpose machines may be used with programswritten in accordance with the teachings herein, or it may be moreconvenient to construct more specialized apparatus (e.g., integratedcircuits) to perform the required method steps. Thus, the invention maybe implemented in one or more computer programs executing on one or moreprogrammable computer systems each comprising at least one processor, atleast one data storage system (including volatile and non-volatilememory and/or storage elements), at least one input device or port, andat least one output device or port. Program code is applied to inputdata to perform the functions described herein and generate outputinformation. The output information is applied to one or more outputdevices, in known fashion.

Each such program may be implemented in any desired computer language(including machine, assembly, or high level procedural, logical, orobject oriented programming languages) to communicate with a computersystem. In any case, the language may be a compiled or interpretedlanguage.

Each such computer program is preferably stored on or downloaded to astorage media or device (e.g., solid state memory or media, or magneticor optical media) readable by a general or special purpose programmablecomputer, for configuring and operating the computer when the storagemedia or device is read by the computer system to perform the proceduresdescribed herein. The inventive system may also be considered to beimplemented as a computer-readable storage medium, configured with acomputer program, where the storage medium so configured causes acomputer system to operate in a specific and predefined manner toperform the functions described herein.

A number of embodiments of the invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention. Forexample, some of the steps described herein may be order independent,and thus can be performed in an order different from that described.

The invention claimed is:
 1. A method for adjusting a level of an audiosignal in an audio processing apparatus, the method comprising: dividingan audio signal into a plurality of frequency bands; delaying at leastone of the plurality of the frequency bands; obtaining modificationparameters for at least one of the plurality of frequency bands, themodification parameters comprising filter coefficients and amplitudescale factors, each of the amplitude scale factors respectivelyoperating in a frequency band of the plurality of frequency bands andeach amplitude scale factor representing an average energy over thefrequency band and a time segment; deriving gain factors for at leastone of the plurality of frequency bands, the gain factors determinedbased on the amplitude scale factors; smoothing the gain factors;determining a level of noise from noise compensation factors; applyingthe gain factors to at least one of the frequency bands to generate gainadjusted frequency bands; adjusting the level of noise based on the gainadjusted frequency bands; filtering at least one of the frequency bandswith a filter generated with the filter coefficients; and synthesizingthe plurality of frequency bands to generate an output audio signal;wherein the gain factors are both time and frequency varying.
 2. Themethod of claim 1, further comprising band smoothing of the amplitudescale factors.
 3. The method of claim 1, wherein the plurality offrequency bands are derived from an analysis filter bank.
 4. The methodof claim 1, wherein the adjusting of the level of noise is performed bya synthesis filterbank.
 5. An audio processing apparatus for adjusting alevel of an audio signal, the audio processing apparatus comprising: ananalysis filterbank for dividing an audio signal into a plurality offrequency bands; a delay unit for delaying at least one of the pluralityof the frequency bands; a parameter generator for obtaining modificationparameters for at least one of the plurality of frequency bands, themodification parameters comprising filter coefficients and amplitudescale factors, each amplitude scale factor respectively operating in afrequency band of the plurality of frequency bands and each amplitudescale factor representing an average energy over the frequency band anda time segment; a first processor for deriving gain factors for at leastone of the plurality of frequency bands, the gain factors determinedbased on the amplitude scale factors; a smoother for smoothing the gainfactors; a second processor for determining a level of noise from noisecompensation factors; a first adjuster for applying the gain factors toat least one of the frequency bands to generate gain adjusted frequencybands; a second adjuster for adjusting the level of noise based on thegain adjusted frequency bands; a filter for filtering at least one ofthe frequency bands, the filter generated with the filter coefficients;and a synthesis filterbank for synthesizing the plurality of frequencybands to generate an output audio signal; wherein the gain factors areboth time and frequency varying.
 6. The audio decoder of claim 5,further comprising a band smoother for smoothing the amplitude scalefactors.
 7. A non-transitory computer readable medium, storing softwareinstructions for controlling a perceptual loudness of a digital audiosignal, which when executed by one or more processors cause performanceof the steps of method claim 1.